Manifest structure must extend general RDF test manifest structure

[gkellogg]

Looking at the existing test manifests (e.g., https://github.com/w3c/web-annotation-tests/blob/c9d3b8b084c509ef9fa7e8ed66e8f9a931348908/framework/annotations/model.test), the form is somewhat ad-hoc. In w3c/web-annotation#290 (comment) I was asked to help create the reports, which as mentioned there, uses the earl-report gem. This expects to find one or more test manifests, which are generally like those described for the RDF tests, for example, the Turtle test manifest. That is, there is an entry of type mf:Manifest, with an entries property referencing a list of entries; the built-in SPARQL query used to access the manifest is the following:

PREFIX mf: <http://www.w3.org/2001/sw/DataAccess/tests/test-manifest#>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

SELECT ?uri ?testAction ?manUri
WHERE {
  ?uri mf:action ?testAction .
  OPTIONAL {
    ?manUri a mf:Manifest; mf:entries ?lh .
    ?lh rdf:first ?uri .
  }
}

But, this can query be overridden if there is some reason to use a different structure. But, to get name and description information for each test, each test entry is expected to have rdfs:label and rdfs:comment entries, in addition to the action. Note, that the names in a JSON-LD version of the manifest can be arbitrary, as long as the context maps them to rdfs:label and rdfs:comment. The mf:action is used to retrieve the test source, and mf:result used to get the expected results, but these aren't strictly necessary, and the query could access some other entry from each test to enumerate all tests.

The earl-report software reads in the manifest and turns it into framed JSON-LD, which is then available to the template.haml which iterates over that to create output for each test. The template will read each mf:action and mf:result file to include in the output, but this isn't necessary. However, it would probably be good to include some information for each test in addition to label and comment, for example we could show each assertion.

The main thing I'm calling for here is to get the test manifests in some normalized form consistent with our reporting needs include a manifest entry with an ordered list for each test including a label and comment using some properties such that this can be transformed to RDF and queried.