forked from w3c/its-2.0-testsuite
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathtest-suite-howto.html
280 lines (225 loc) · 17.8 KB
/
test-suite-howto.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8"/>
<title>Test Suite How-To Guide</title>
<style type="text/css"></style>
</head>
<body>
<h1><span>Test Suite How-To Guide</span></h1>
<p><span>There are three parts to the test suite:</span></p>
<ol>
<li><span><a href="#1">Conformance testing for HTML & XML</a></span></li>
<li><span><a href="#2">Testing for NIF</a></span></li>
<li><span><a href="#3">Input file validation</a></span></li>
</ol>
<h2><span><a name="1"></a>1. Conformance testing for HTML & XML</span></h2>
<p><span>This conformance testing is used to determine whether ITS 2.0 implementations meet the
ITS 2.0 specification standard. The test suite has a set of test documents for both XML and
HTML which are then used to validate the different ITS 2.0 constructs available for each
data category. There are a total of 225 test files spread across all 19 data categories for
both HTML and XML.</span></p>
<h3><span>1.1 How to use the test suite?</span></h3>
<p><img height="301" src="testsuite-visualization.png" width="483" alt="test suite output format" />
<br/>
<span><b>Figure 1: Testsuite processing 	Input to Output</b></span></p>
<p><span>The validation of the input files is done through processing the input file so that it
outputs a file to match a corresponding gold standard output file. The gold standard was
develop to be easy to understand 	and implement for conformance testers. The gold
standard output format was developed by the ITS 2.0 working group. Figure 1 above describes
the test suite files process.</span></p>
<h3><span>1.2 Gold Standard output specifications details:</span></h3>
<p><span>	The gold standard output files have the following characteristics:</span></p>
<ol>
<li><span>Every element and attribute path from the XML and HTML file are listed (apart from
the content within script elements in HTML)</span></li>
<li><span>The output has to be tab-delimited
format:								</span>
<p><span class="c3"><i>/html/body[1]/p[1]/span[2]								annotatorsRef="text-analysis|http://enrycher.ijs.si"								taConfidence="0.5"								taIdent="301467919"								taSource="Wordnet3.0"</i></span></p></li>
<li><span>The attributes within elements have
to be in alphabetical order:</span>
<p><span class="c3"><i>/doc/header[1]/its:rules[1]/its:locQualityIssueRule[2]								</i></span></p>
<p><span class="c3"><i>/doc/header[1]/its:rules[1]/its:locQualityIssueRule[2]/@locQualityIssueComment</i></span></p>
<p><span class="c3"><i>/doc/header[1]/its:rules[1]/its:locQualityIssueRule[2]/@locQualityIssueProfileRef</i></span></p>
<p><span class="c3"><i>/doc/header[1]/its:rules[1]/its:locQualityIssueRule[2]/@locQualityIssueSeverity</i></span></p>
<p><span class="c3"><i>/doc/header[1]/its:rules[1]/its:locQualityIssueRule[2]/@locQualityIssueType</i></span></p>
<p><span class="c3"><i>/doc/header[1]/its:rules[1]/its:locQualityIssueRule[2]/@locQualityIssuesRef</i></span></p>
<p><span class="c3"><i>/doc/header[1]/its:rules[1]/its:locQualityIssueRule[2]/@selector</i></span></p></li>
<li><span>The rules output also have to be in alphabetical orders:</span>
<p><span class="c3 c12 c8">/html/body[1]/p[1]/span[2]								annotatorsRef="text-analysis|http://enrycher.ijs.si"								taConfidence="0.5"								taIdent="301467919"								taSource="Wordnet3.0"</span></p></li>
<li><span>The rules output can't have Pointer in it but it has to have its equivalent
(unless it is in the target pointer data category then it must display
targetpointer="...." and the pointer
details):								</span>
<p><span><b>Incorrect:</b></span></p>
<p><span><i>/doc/para[1]/issue[2]
								locQualityIssueTypePointer="misspelling"								locQualityIssuesRefPointer="#l1234"								</i></span></p>
<p><span><b>Correct:</b></span><span class="c4"
>								</span></p>
<p><span class="c3"><i>/doc/para[1]/issue[2]								locQualityIssueType="misspelling"								locQualityIssuesRef="#l1235"								</i></span></p></li>
<li><span>The rules output for local html rules have to be like their global
counterparts:</span>
<p><span><b>Incorrect:</b></span></p>
<p><span><i>/doc/p[1]
								its-loc-quality-issue-type="misspelling"								locQualityIssuesRef="#l1235"								</i></span></p>
<p><span class="c4 c6"><b>Correct:								</b></span></p>
<p><span><i>/doc/p[1]
								locQualityIssueType="misspelling"
								locQualityIssuesRef="#l1235"								</i></span></p>
</li>
</ol>
<h3><span>1.3 How gold standard output is compared to implementors output?</span></h3>
<p><span>The output of the implementors is located in the folder</span>
<span>ITS-2.0-Testsuite/its2.0/</span><span class="c4 c8 c6">outputimplementors</span>
<span>which</span><span class="c4 c6">	</span><span>can be tested against the gold
standard output files located in the</span>
<span class="c4 c6">ITS-2.0-Testsuite/its2.0/expected</span>
<span>folder. This can be done simply through doing a diff of the implementors output files
and the corresponding gold standard output files. To automate this process for implementors
a test suite dashboard was created which did the following tasks :</span></p>
<ol>
<li><span>Help to track the process of implementers in relation to the tests in which they
were committed to complete</span><span class="c4 c22 c6">	(indicated on the dashboard
via</span>
<span class="c4 c12 c22 c8 c6">N/A = the implementer did not commit to run the
test)</span><span>.</span></li>
<li><span>Help to track if the implementors output file matches the corresponding gold
standard output file</span><span class="c4 c12 c22 c8 c6">	(indicated on the
dashboard via fnf: the output file from the implementer has not been found OK = the output
file is identical to the reference output file)</span><span>.</span></li>
<li><span>Help to track if the output file that is in the output folder for a particular tests
doesn't match the corresponding gold standard output file</span><span>	</span><span
class="c4 c12 c22 c8 c6">(indicated on the dashboard via error = an error occurred, e.g.
the output file is not available or it is not identical to the reference output file. Move
the mouse over error to see details)</span><span>.</span></li>
<li><span>Help to track if the implementor has not committed an output file for a
corresponding test</span>
<span class="c4 c12 c8 c6 c22">(indicated on the dashboard via fnf: the output file from the
implementer has not been found)</span><span>.</span></li>
<li><span>The dashboard can also track how many tests a particular implementor has left to
run.</span></li>
</ol>
<h3><span>1.4</span>
<span>Validating Output Test Files</span></h3>
<p><span>To validate the implementors output files the testsuite dashboard has to be compiled so
that a diff across all files in the</span>
<span>ITS-2.0-Testsuite/its2.0/outputimplementors</span>
<span class="c4 c8">	folder can be done</span><span>	</span><span class="c4 c8"
>against the corresponding files in the 	gold standard output</span>
<span>ITS-2.0-Testsuite/ITS2.0/expected</span><span class="c4 c8">	 folder. The test
suite dashboard can be compiled by doing the following:</span></p>
<ol>
<li><span>Download saxon.jar from here:</span>
<span><a href="https://dl.dropbox.com/u/65779171/saxon.jar"
>https://dl.dropbox.com/u/65779171/saxon.jar</a></span></li>
<li><span>Then use this command (Linux/Mac/Windows): java -jar /path/of/file/saxon.jar
testsuiteMaster.xml testsuiteDashboard.xsl -o:testSuiteDashboard.html</span></li>
<li><span>Upload newly compiled testsuiteDashboard.html to the git hub</span></li>
<li><span>Check the state of your files in the related data categories on this web
page:</span>
<span><a class="c15"
href="http://htmlpreview.github.com/?https://raw.github.com/finnle/ITS-2.0-Testsuite/master/its2.0/testSuiteDashboard.html">
http://htmlpreview.github.com/?https://raw.github.com/finnle/ITS-2.0-Testsuite/master/its2.0/testSuiteDashboard.html</a></span></li>
</ol>
<p><span>The files for the test suite dashboard are as follows:</span></p>
<ol>
<li><span>testsuiteMaster.xml -</span>
<span>has a list of the implementer's and the tests that they are involved in and aids in
the creation of the testSuiteDashboard.html</span></li>
<li><span>testsuiteDashboard.xsl -</span>
<span>does the diffs between the implementer's output and the gold standard
output.</span></li>
<li><span>testSuiteDashboard.xml -</span>
<span>gives information on the errors between the diff's for the implementors output and
the gold standard output</span></li>
<li><span>testSuiteDashboard.html -</span><span>	the html test suite dashboard which is
located live</span>
<span><a
href="http://htmlpreview.github.io/?https://raw.github.com/finnle/ITS-2.0-Testsuite/master/its2.0/testSuiteDashboard.html"
> here</a></span><span>	</span></li>
</ol>
<h2><span><a name="2"></a>2. Testing for</span>
<span class="c2">NIF</span><span>	</span><span>	</span></h2>
<p>This part of the test suite is not normative. The aim is to support implementers in working with the
non normative <a class="c15" href="http://www.w3.org/TR/its20/#conversion-to-nif">Conversion to NIF</a> algorithm.
	</p>
<p>The
NIF test suite has a set of test documents for HTML for the
<span>Localisation Quality Issue</span><span class="c4">	data category which are then
used to validate the different aspects of ITS 2.0 and NIF against various constructs
available to the Localisation Quality Issue data category. There are a total of 11 files
only for HTML in the Localisation Quality Issue data category. The input files are located
in the</span><span class="c4">	</span>
<span class="c4 c6">ITS-2.0-Testsuite/its2.0/nif-conversion/input</span>
<span class="c4">folder.</span></p>
<p><span>The validation of the input files is done through processing the input file so that it
outputs a file to match a gold standard output file. The gold standard for NIF is RDF output
using the ITS 2.0 ontology. The gold standard output for NIF can be reached by following the
NIF conversion algorithm discussed in the ITS 2.0 specification located</span>
<span><a class="c15" href="http://www.w3.org/TR/its20/#conversion-to-nif">here</a></span><span
class="c4">.</span></p>
<h3><span>2.1 How gold standard 	NIF output is compared to implementors output?</span></h3>
<p><span>The NIF output files are compared via the use of SPARQL queries done over the
implementers RDF/NIF output files (.ttl). If the SPARQL queries are successful then the NIF
output files are correct and meet the gold standard. The implementer's NIF test output
files are located in the ITS-2.0-Testsuite/its2.0/nif-conversion/expected folder.</span></p>
<h3><span>2.2 Validating NIF output files</span></h3>
<p><span>Prerequisites:</span><span>	Java and Unix Shell</span></p>
<ol>
<li><span>create a temporary folder for output files (hence called $datafolder)</span></li>
<li><span>read ITS files from "its2.0/nif-conversion/input/" one by one, convert to NIF and
write output files in turtle to $datafolder</span></li>
<li><span>go to directory cd its2.0/nif-conversion/sparqltest</span></li>
<li><span>run : ./executeTests.sh ../relative/pathTo/$datafolder</span></li>
</ol>
<p><span>Explanations of output:</span></p>
<ol>
<li><span>If no message appears between "Running: test1.sparql" and "Done: test1.sparql" the
test was successful.</span></li>
<li><span>Otherwise the output filename and additional debug output is shown</span></li>
</ol>
<h2><span><a name="3"></a>3. Input file Validation</span></h2>
<p><span>This part of the test suite is vital and it is the testing of the HTML and XML input
files to see if they are 	valid in accordance to the ITS 2.0 schema specification for
XML and HTML. More information about this validator can be found</span>
<span><a href="http://validator.nu/?doc=http%3A%2F%2Fwww.w3.org%2FTR%2F2012%2FWD-its20-20121206%2Fexamples%2Fhtml5%2FEX-term-html5-local-1.html&schema=http%3A%2F%2Fs.validator.nu%2Fhtml5-its.rnc+http%3A%2F%2Fs.validator.nu%2Fhtml5%2Fassertions.sch+http%3A%2F%2Fc.validator.nu%2Fall%2F"> here</a></span><span>	and</span>
<span><a href="http://about.validator.nu/">also here</a></span><span>.</span></p>
<h3><span>3.1 Validating Input Test Files</span></h3>
<p><span>The following sections detail how to validate the test suite input files both HTML and
XML.</span></p>
<h3><span>3.1.1 Validating XML test files</span></h3>
<ol>
<li><span>Download and install Ant from</span>
<span><a href="http://ant.apache.org/">http://ant.apache.org/</a></span></li>
<li><span>Run 'ant validate-xml' command in its2.0 directory</span></li>
</ol>
<h3><span>3.1.2 Validating HTML test files</span></h3>
<ol>
<li><span>Download and install Ant from</span>
<span><a href="http://ant.apache.org/">http://ant.apache.org/</a></span></li>
<li><span>Download html5-its-tools from</span>
<span><a href="https://github.com/kosek/html5-its-tools"
>https://github.com/kosek/html5-its-tools</a></span></li>
<li><span>Modify its2.0/build.properties to point to your local copy of
html5-its-tools</span></li>
<li><span>Run 'ant validate-html' command in its2.0 directory</span></li>
</ol>
<h3><span>3.1.3 Validating all test files</span></h3>
<ol>
<li><span>Make sure that XML and HTML validation described above works for you</span></li>
<li><span>Run 'ant' command in its2.0 directory</span></li>
<li><span>Please note that HTML schema doesn't supports RDFa so RDFa attributes are reported
as errors</span></li>
<li><span>Please note that currently Schematron validation is not performed so some errors are
not detected</span></li>
</ol>
<h2><span>4. Xliff Sample</span></h2>
<p><span>In addition to conformance testing and validation of XML and HTML files it
provides</span>
<span>XLIFF samples</span><span>. These are not used in conformance testing, but demonstrate
the representation of ITS 2.0. metadata in XLIFF. These XLIFF files are located
in</span><span>	</span><span class="c6">ITS-2.0-Testsuite</span><span>/</span><span
class="c6">its2.0/</span><span>xliffsamples</span>
<span>folder of the github.</span></p>
</body>
</html>