Adds text descriptions for intended system macro test cases (#128)

---------

Co-authored-by: Zack Slayton <zack.slayton@gmail.com>
Co-authored-by: Tyler Gregg <greggt@amazon.com>

conformance/README.md

@@ -3,8 +3,53 @@
 This directory contains conformance tests for Ion, expressed using a small,
 declarative domain-specific language.
 
+**Contents**
+ * `core/` – test cases for verifying the functionality of a conformance DSL runner
+ * `data_model/` – test cases for the encodings of the Ion data model
+ * `eexp/` – test cases for parsing/interpretation of e-expressions and their arguments
+ * `ion_encoding/` – test cases for encoding directives
+   * `module/`
+ * `system_macros/` – test cases for each of the system macros
+
+> [!WARNING]
 > This test suite and its DSL are **Work In Progress**.
 
+
+# Test Tiers
+
+The full DSL requires a complete Ion text parser at minimum (to read
+the DSL S-expressions), and some aspects require a writer or encoder as well.
+This raises the question of how we can lower the barrier to entry so that we can write tests
+that can be exercised for an in-progress Ion implementation earlier than that
+point.
+
+We intend to approach this problem by carefully partitioning the test files,
+and by expressing the earliest phases (the lowest-level test cases) in a subset of Ion that
+can be safely converted to JSON using `ion to json` (see [`ion-cli`](https://github.com/amazon-ion/ion-cli)).
+For example,
+
+```
+["ion_1_1", ["each",
+               ["text", "1"],
+               ["bytes", 0x61, 0x01],
+               ["toplevel", 1],
+               ["produces", 1]]]
+```
+
+The next logical tiers would introduce basic symbols and sexps into the parser.
+Here, the `denotes` expectation becomes valuable:
+
+```
+["ion_1_x", ["text", "(1 2.3)"],
+            ["denotes", ["Sexp", ["Int", 1], ["Decimal", 23, -1]]]]
+```
+
+The inner clauses should align with the formal data model in the denotational
+semantics, which would (more or less) reduce the components to integers and
+strings.
+
+# The Test DSL
+
 At a high level, each test case in the DSL consists of an input document and an
 expectation that should be met when the document is evaluated by an Ion
 implementation.  Each expectation is either a series of application values that
@@ -736,39 +781,6 @@ in terms of Unicode code points, which is needed to test parsing of escape
 sequences.
 
 
-# Test Tiers
-
-The full DSL requires a complete Ion text parser at minimum (to read
-the DSL S-expressions), and some aspects require a writer or encoder as well.
-This raises the question of how we can lower the bar so that we can write tests
-that can be exercised for an in-progress Ion implementation earlier than that
-point.
-
-We intend to approach this problem by carefully partitioning the test files,
-and by expressing the earliest phases, that is the lowest-level test cases,
-in JSON. For example:
-
-```
-["ion_1_1", ["each",
-               ["text", "1"],
-               ["bytes", 0x61, 0x01],
-               ["toplevel", 1],
-               ["produces", 1]]]
-```
-
-The next logical tiers would introduce basic symbols and sexps into the parser.
-Here, the `denotes` expectation becomes valuable:
-
-```
-["ion_1_x", ["text", "(1 2.3)"],
-            ["denotes", ["Sexp", ["Int", 1], ["Decimal", 23, -1]]]]
-```
-
-The inner clauses should align with the formal data model in the denotational
-semantics, which would (more or less) reduce the components to integers and
-strings.
-
-
 # WIP TODOs
 
 * The denotational test cases are wired with a small catalog of shared symtabs

conformance/grammar.isl

@@ -1,3 +1,6 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
 $ion_schema_2_0
 
 type::{

conformance/system_macros/add_macros.ion

@@ -0,0 +1,9 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+// Test Cases:
+// add_macros can be invoked using any type of macro reference.
+// the argument values may be zero or more macro definitions
+// add_macros does not have any side-effects on the symbol table
+// add_macros should append its arguments to the macro table of the default module
+// annotations on any argument value should signal an error (because it's not allowed in the macro table rather than because of any specific validation by the add_macros macro)

conformance/system_macros/add_symbols.ion

@@ -0,0 +1,9 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+// Test Cases:
+// add_symbols can be invoked using any type of macro reference.
+// the argument values may be zero or more non-null text values
+// add_symbols does not have any side-effects on the macro table
+// add_symbols should append its arguments to the symbol table of the default module
+// annotations on any argument value should signal an error (because it's not allowed in the symbol table rather than because of any specific validation by the add_symbols macro)

conformance/system_macros/annotate.ion

@@ -0,0 +1,9 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+// Test Cases:
+// annotate can be invoked using any type of macro reference.
+// annotate adds annotations to a value
+// if the value has existing annotations, they are preserved and annotate prepends annotations to the existing annotations of the value
+// the first argument must be zero or more non-null text values ($0 is allowed)
+// the second argument must be a single expression that also expands to a single expression

conformance/system_macros/default.ion

@@ -0,0 +1,10 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+// Test Cases:
+// default can be invoked using any type of macro reference.
+// both arguments may be any expression or expression group
+// if the first argument is non-empty
+//    - `default` produces the first argument as its result
+//    - the second argument is not expanded
+// if the first argument is empty (i.e. none), `default` produces the second argument as its result

conformance/system_macros/delta.ion

@@ -0,0 +1,12 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+// Test Cases:
+// delta can be invoked using any type of macro reference.
+// the arguments of delta must be non-null integers
+// the "initial" argument is required
+// the "delta" argument can be zero or more values
+// any annotations on the arguments are silently dropped
+// the output of delta is defined as follows:
+//    output₀ = initial + delta₀
+//    outputₙ₊₁ = outputₙ + deltaₙ₊₁

conformance/system_macros/flatten.ion

@@ -0,0 +1,8 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+// Test Cases:
+// flatten can be invoked using any type of macro reference.
+// the argument values may be zero or more non-null lists and s-expressions
+// flatten expands to a stream whose elements are the concatenated elements of all its arguments
+// annotations on the argument (list and sexp) values are silently dropped

conformance/system_macros/make_blob.ion

@@ -0,0 +1,8 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+// Test Cases:
+// make_blob can be invoked using any type of macro reference.
+// make_blob creates a single unannotated blob from zero or more lob values
+// null values and non-lob values must signal an error
+// annotations on the argment values are silently dropped

conformance/system_macros/make_decimal.ion

@@ -0,0 +1,9 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+// Test Cases:
+// make_decimal can be invoked using any type of macro reference.
+// make_decimal creates a single unannotated decimal
+// the coefficient argument must be a non-null integer
+// the exponent argument must be a non-null integer
+// annotations on the argument values are silently dropped

conformance/system_macros/make_field.ion

@@ -0,0 +1,10 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+// Test Cases:
+// make_field can be invoked using any type of macro reference.
+// the first argument must expand to a single non-null text value (`$0` is allowed)
+// annotations first argument are silently dropped
+// the second argument may be any single expression that expands to a single expression
+// make_field expands to a single unannotated struct containing only field(s) with the first argument as the field name,
+//     and the second argument value as the field value.

conformance/system_macros/make_list.ion

@@ -0,0 +1,8 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+// Test Cases:
+// make_list can be invoked using any type of macro reference.
+// the argument values may be zero or more non-null lists and s-expressions
+// make_list creates a single unannotated list whose elements are the concatenated elements of all its arguments
+// annotations on the argument values are silently dropped

conformance/system_macros/make_sexp.ion

@@ -0,0 +1,8 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+// Test Cases:
+// make_sexp can be invoked using any type of macro reference.
+// the argument values may be zero or more non-null lists and s-expressions
+// make_sexp creates a single unannotated s-expression whose elements are the concatenated elements of all its arguments
+// annotations on the argument values are silently dropped

conformance/system_macros/make_string.ion

@@ -0,0 +1,8 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+// Test Cases:
+// make_string can be invoked using any type of macro reference.
+// make_string creates a single unannotated string from zero or more text values
+// null values, unknown symbol text, and non-text values must signal an error
+// annotations on the argment values are silently dropped

conformance/system_macros/make_struct.ion

@@ -0,0 +1,8 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+// Test Cases:
+// make_struct can be invoked using any type of macro reference.
+// the argument values may be zero or more non-null structs
+// make_struct creates a single unannotated struct whose fields are the concatenated fields of all its arguments
+// annotations on the argument values are silently dropped

conformance/system_macros/make_symbol.ion

@@ -0,0 +1,8 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+// Test Cases:
+// make_symbol can be invoked using any type of macro reference.
+// make_symbol creates a single unannotated symbol from zero or more text values
+// null values, unknown symbol text, and non-text values must signal an error
+// annotations on the argment values are silently dropped

conformance/system_macros/make_timestamp.ion

@@ -0,0 +1,22 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+// Test Cases:
+// make_timestamp can be invoked using any type of macro reference.
+// make_timestamp creates a single unannotated timestamp
+// any annotations on the argment values are silently dropped
+// if any argument is null, it should signal an error
+// year must be from 0 to 9999 (inclusive)
+// month must be from 1 to 12 (inclusive)
+// if month is present, year must also be present
+// day must be from 1 to 28,29,30, or 31 (inclusive) depending on the month
+// if day is present, month must also be present
+// hour must be from 0 to 23 (inclusive)
+// if hour is present, minute and day must also be present
+// minute must be from 0 to 59 (inclusive)
+// if minute is present, hour must also be present
+// second must be a decimal value from +0 to 60 (exclusive) with any precision
+// if second is present, minute must also be present
+// offset minutes argument may not be present unless minutes is present.
+// if offset minutes is present, it must be in the range -1440 to 1440 (inclusive)
+// if offset minutes is absent, the resulting timestamp has unknown offset.

conformance/system_macros/meta.ion

@@ -0,0 +1,6 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+// Test Cases:
+// meta can be invoked using any type of macro reference.
+// meta has one parameter that accepts zero-to-many argument expressions and always produces nothing

conformance/system_macros/none.ion

@@ -0,0 +1,6 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+// Test Cases:
+// none can be invoked using any type of macro reference.
+// none accepts no arguments and produces nothing

conformance/system_macros/parse_ion.ion

@@ -0,0 +1,25 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+(ion_1_1 "parse_ion can be invoked"
+          (each "in text with an unqualified macro name"
+                (text ''' (:parse_ion "true") ''')
+                "in text with an unqualified macro address"
+                (text ''' (:16 "true") ''')
+                "in text with a qualified macro name"
+                (text ''' (:$ion::parse_ion "true") ''')
+                "in text with a qualified macro address"
+                (text ''' (:$ion::16 "true") ''')
+                "in binary with a system macro address"
+                (binary "EF 10 94 74 72 75 65")
+                "in binary with a user macro address"
+                (binary "10 94 74 72 75 65")
+                (produces true)))
+
+// parse_ion can read Ion 1.0 and Ion 1.1, text and binary
+
+// parse_ion always produces user values. See https://github.com/amazon-ion/ion-docs/issues/365 for complications to be resolved.
+
+// parse_ion requires exactly one argument
+
+// parse_ion argument must be a string or lob literal

conformance/system_macros/repeat.ion

@@ -0,0 +1,8 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+// Test Cases:
+// repeat can be invoked using any type of macro reference.
+// repeat can repeat things the correct number of times
+// the number of repetitions must be a single expression that expands to one positive integer (i.e. an integer greater than zero)
+// the thing to be repeated must be at least one expression of any type

conformance/system_macros/set_macros.ion

@@ -0,0 +1,10 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+// Test Cases:
+// set_macros can be invoked using any type of macro reference.
+// the argument values may be zero or more macro definitions
+//    TODO: Could it also be a module or an export?
+// set_macros does not have any side-effects on the symbol table
+// set_macros replaces the current macro table with a new macro table that consists of the arguments of set_macros
+// annotations on any argument value should signal an error (because it's not allowed in the macro table rather than because of any specific validation by the set_macros macro)

conformance/system_macros/set_symbols.ion

@@ -0,0 +1,9 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+// Test Cases:
+// set_symbols can be invoked using any type of macro reference.
+// the argument values may be zero or more non-null text values
+// set_symbols does not have any side-effects on the macro table
+// set_symbols replaces the current symbol table with a new symbol table that consists of the arguments of set_symbols
+// annotations on any argument value should signal an error (because it's not allowed in the symbol table rather than because of any specific validation by the set_symbols macro)

conformance/system_macros/sum.ion

@@ -0,0 +1,8 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+// Test Cases:
+// sum can be invoked using any type of macro reference.
+// the arguments of sum must be zero or more non-null integers
+// any annotations on the arguments are silently dropped
+// the output of sum is the sum of all its argument values

conformance/system_macros/use.ion

@@ -0,0 +1,12 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+// Test Cases:
+// `use` can be invoked using any type of macro reference.
+// the first parameter must have a single argument value that is a non-null, unannotated string
+// the second parameter may be absent or it may be a single value that is a non-null, unannotated, positive integer
+// if the second parameter is absent, it has an implicit default value of 1
+// if the <catalog_key, version> pair cannot be located, in the catalog, the reader should signal an error
+// invoking `use` should preserve all existing symbols and macros
+// invoking `use` should import the specified module and append its content to the default module's symbol table and macro table
+// `use` should not (re)define any existing stream-level module bindings

conformance/system_macros/values.ion

@@ -0,0 +1,6 @@
+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+// Test Cases:
+// values can be invoked using any type of macro reference.
+// values has one parameter that accept zero-to-many arguments and produces its input