Detailed Results of PSO

Table 1. Sizes of constrained covering arrays generated by different constraint handlers

	Verify	Solver	Tolerate	Replace	P-value
Apache	42.93	43.03	43.47	37.8	<2e-16
Bugzilla	23.07	23.03	23.1	19.43	1.3e-14
GCC	27.27	27.17	28.07	23.13	<2e-16
SPIN-S	27.57	27.9	27.67	28.13	0.20090
SPIN-V	45.6	45.9	46.17	42.9	<2e-16
M1	54.8	54.47	54.47	50.47	<2e-16
M2	40.8	40.67	41.33	34.37	<2e-16
M3	21.27	21.07	20.7	19.7	3.6e-07
M4	29.03	29.03	29.03	24.73	<2e-16
M5	68.47	68.43	65.93	59.63	<2e-16
M6	34.13	33.83	33.77	27.93	<2e-16
M7	12.2	12.37	12.23	10.87	<2e-16
M8	58	57.7	57.87	51.53	<2e-16
M9	25.9	26.2	25.2	21.17	<2e-16
M10	64.03	63.8	61.7	56.1	<2e-16
M11	59.5	59.17	59	54.33	<2e-16
M12	55.03	55.23	55.67	47.77	<2e-16
M13	48.63	48.3	49.2	41.63	<2e-16
M14	52.7	51.93	52.73	45.33	<2e-16
M15	40.83	40.7	41.2	36.9	<2e-16
M16	32.97	33.4	33.3	27.13	<2e-16
M17	54.37	55	55.1	47	<2e-16
M18	59.93	60.03	58.77	53.37	<2e-16
M19	70.07	70.27	66.83	58.9	<2e-16
M20	78.53	78.6	72.03	68.77	<2e-16
M21	50.13	50.4	51.97	44.7	<2e-16
M22	44.67	45.43	45.3	36.57	<2e-16
M23	16	15.93	16.07	14	<2e-16
M24	61.13	60.83	59.77	55.33	<2e-16
M25	68	67.67	66.8	62.8	<2e-16
M26	42.73	43.03	42.93	38.73	<2e-16
M27	48.97	48.33	48.5	40.83	<2e-16
M28	73.7	72.9	70.03	64.03	<2e-16
M29	39.73	38.93	39.67	31.47	<2e-16
M30	26.4	26.47	27.2	22.4	<2e-16

Table 2. Computational cost (seconds) of the generation algorithm with different constraint handlers

	Verify	Solver	Tolerate	Replace	P-value
Apache	34.27	245.5	25.12	24.3	<2e-16
Bugzilla	1.13	14.28	0.96	0.88	<2e-16
GCC	43.04	566.86	22.11	19.97	<2e-16
SPIN-S	0.22	6.7	0.2	0.15	<2e-16
SPIN-V	19.95	282.13	3.96	2.63	<2e-16
M1	23.06	375.31	10.61	11.41	<2e-16
M2	13.12	199.29	6.79	6.85	<2e-16
M3	0.43	11.01	0.29	0.27	<2e-16
M4	2.66	68.48	1.63	1.61	<2e-16
M5	85.07	1124.41	48.48	59	<2e-16
M6	4.38	135.69	3.11	3.27	<2e-16
M7	0.2	6.34	0.18	0.17	<2e-16
M8	30.59	528.18	17.67	17.95	<2e-16
M9	5.22	78.2	3.07	4.95	<2e-16
M10	67.65	807.38	32.74	38.71	<2e-16
M11	23.5	418.86	12.18	12.31	<2e-16
M12	36.8	697.64	24.48	25.27	<2e-16
M13	25.58	531.27	16.67	17.23	<2e-16
M14	11.12	306.33	8.35	8.04	<2e-16
M15	7.11	99.41	3.08	2.66	<2e-16
M16	5.99	163.49	4.27	4.32	<2e-16
M17	48.02	558.67	22.42	24.58	<2e-16
M18	43.01	733.94	26.64	26.76	<2e-16
M19	182.22	1459.63	75.68	101.16	<2e-16
M20	70.33	1044.63	47.59	52.43	<2e-16
M21	8.93	246.09	6.62	6.91	<2e-16
M22	7.15	180.19	4.69	4.64	<2e-16
M23	0.32	11.44	0.21	0.25	<2e-16
M24	30.51	571.35	17.66	18.37	<2e-16
M25	51.2	698.75	28.2	30.82	<2e-16
M26	18.5	300.6	8.23	8.06	<2e-16
M27	6.13	112.6	3.54	3.21	<2e-16
M28	144.44	1527.83	73.52	94.42	<2e-16
M29	31.95	482.01	16.36	16.55	<2e-16
M30	5.16	137.68	2.88	2.79	<2e-16

Table 3. Proportions of failures detected by covering arrays generated by different constraint handlers

	k = 3					k = 4					k = 5					k = 6
	Verify	Solver	Tolerate	Replace	P-value	Verify	Solver	Tolerate	Replace	P-value	Verify	Solver	Tolerate	Replace	P-value	Verify	Solver	Tolerate	Replace	P-value
Apache	0.98	0.98	0.99	0.98	0.09658	0.89	0.89	0.9	0.87	0.00083	0.66	0.65	0.66	0.62	0.00846	0.43	0.41	0.42	0.39	0.04063
Bugzilla	0.95	0.94	0.95	0.94	0.0075	0.73	0.74	0.74	0.68	5e-05	0.47	0.46	0.49	0.41	2.5e-09	0.26	0.26	0.25	0.22	0.00095
GCC	0.98	0.97	0.98	0.95	5.9e-06	0.82	0.81	0.8	0.77	0.00012	0.53	0.54	0.56	0.48	4.7e-06	0.33	0.32	0.33	0.28	0.00029
SPIN-S	0.87	0.88	0.87	0.88	0.83108	0.56	0.58	0.57	0.58	0.32991	0.33	0.34	0.32	0.33	0.28177	0.17	0.18	0.18	0.17	0.88351
SPIN-V	0.94	0.94	0.94	0.94	0.54138	0.72	0.73	0.73	0.74	0.4753	0.51	0.52	0.51	0.49	0.07865	0.28	0.28	0.29	0.27	0.14099
M1	0.99	0.98	0.99	0.98	0.03076	0.88	0.88	0.88	0.85	8e-05	0.67	0.65	0.66	0.65	0.47828	0.46	0.48	0.46	0.43	0.001
M2	0.98	0.98	0.98	0.97	0.03265	0.86	0.85	0.86	0.82	1e-05	0.62	0.63	0.62	0.56	1e-09	0.39	0.39	0.38	0.37	0.29973
M3	0.89	0.91	0.9	0.89	0.01007	0.71	0.7	0.71	0.69	0.26453	0.45	0.44	0.44	0.42	0.28735	0.25	0.25	0.26	0.24	0.79427
M4	0.96	0.94	0.95	0.94	0.00168	0.81	0.8	0.81	0.76	4.5e-07	0.56	0.54	0.55	0.49	4.1e-07	0.32	0.32	0.32	0.29	0.01505
M5	0.99	0.99	0.99	0.99	0.39932	0.95	0.94	0.93	0.92	2e-05	0.76	0.75	0.75	0.71	5e-08	0.52	0.51	0.48	0.46	6e-05
M6	0.98	0.98	0.98	0.98	0.20271	0.9	0.9	0.89	0.84	3.9e-15	0.67	0.68	0.67	0.61	1.3e-07	0.47	0.48	0.46	0.39	3.8e-12
M7	0.88	0.9	0.88	0.86	2e-05	0.66	0.66	0.66	0.63	0.01889	0.4	0.39	0.38	0.37	0.05475	0.24	0.21	0.22	0.21	0.02173
M8	0.99	0.99	0.99	0.98	0.31143	0.92	0.92	0.91	0.9	0.01358	0.72	0.72	0.7	0.67	2.8e-06	0.48	0.51	0.48	0.47	0.00457
M9	0.96	0.96	0.95	0.93	7.2e-07	0.82	0.83	0.83	0.77	1.6e-08	0.66	0.66	0.63	0.58	1.7e-12	0.42	0.43	0.4	0.37	3.4e-06
M10	0.99	0.98	0.98	0.98	0.00205	0.92	0.91	0.91	0.9	0.02984	0.74	0.75	0.73	0.7	3.1e-06	0.5	0.5	0.51	0.47	0.0009
M11	0.98	0.98	0.98	0.98	0.20764	0.89	0.89	0.88	0.87	0.00085	0.7	0.7	0.7	0.67	9e-05	0.44	0.45	0.46	0.43	0.06222
M12	0.98	0.99	0.98	0.98	0.01326	0.91	0.92	0.91	0.89	2e-05	0.72	0.74	0.73	0.66	6e-10	0.53	0.52	0.52	0.47	6.9e-06
M13	0.99	0.99	0.99	0.98	0.05306	0.92	0.91	0.91	0.87	1.2e-08	0.71	0.71	0.71	0.66	2e-05	0.47	0.46	0.46	0.42	0.00325
M14	0.98	0.98	0.99	0.98	0.40368	0.89	0.9	0.88	0.86	0.00013	0.68	0.66	0.66	0.61	4.5e-07	0.45	0.45	0.43	0.37	1.4e-09
M15	0.97	0.96	0.96	0.96	0.43058	0.84	0.83	0.83	0.81	0.08955	0.59	0.57	0.58	0.56	0.04337	0.36	0.37	0.38	0.35	0.19785
M16	0.98	0.98	0.98	0.96	1e-05	0.87	0.87	0.88	0.83	1.4e-07	0.71	0.71	0.69	0.63	4.4e-11	0.46	0.47	0.47	0.42	4.1e-06
M17	0.99	0.99	0.99	0.99	0.4963	0.91	0.91	0.91	0.9	0.11989	0.73	0.76	0.76	0.69	1.9e-10	0.5	0.51	0.49	0.42	5e-11
M18	0.99	0.99	0.99	0.99	0.63035	0.9	0.89	0.89	0.87	0.00382	0.7	0.71	0.71	0.65	5.1e-06	0.43	0.43	0.44	0.41	0.1964
M19	0.99	0.99	0.99	0.99	0.36541	0.93	0.94	0.93	0.92	0.00038	0.77	0.75	0.72	0.7	5e-08	0.53	0.52	0.5	0.48	0.00021
M20	0.99	0.99	0.99	0.99	0.18258	0.94	0.93	0.9	0.91	1.2e-06	0.77	0.77	0.74	0.73	1.1e-06	0.57	0.58	0.53	0.53	3.5e-06
M21	0.99	0.99	0.99	0.99	0.68737	0.93	0.93	0.93	0.91	0.00098	0.77	0.77	0.75	0.73	2e-05	0.54	0.55	0.56	0.51	0.00242
M22	0.98	0.98	0.98	0.98	0.01039	0.89	0.89	0.9	0.85	4e-09	0.7	0.72	0.7	0.63	3.6e-12	0.47	0.49	0.47	0.4	5.4e-11
M23	0.9	0.9	0.9	0.88	0.00183	0.67	0.66	0.67	0.64	0.02968	0.43	0.43	0.44	0.38	0.00014	0.24	0.24	0.25	0.21	0.00444
M24	0.99	0.99	0.99	0.99	0.52654	0.92	0.92	0.91	0.91	0.16972	0.75	0.76	0.74	0.71	0.00075	0.52	0.52	0.53	0.46	7.7e-07
M25	0.99	0.99	0.99	0.99	0.52693	0.89	0.88	0.87	0.88	0.08859	0.71	0.69	0.69	0.69	0.22688	0.47	0.48	0.46	0.47	0.43201
M26	0.98	0.98	0.98	0.97	0.00021	0.88	0.88	0.87	0.86	0.00037	0.67	0.67	0.68	0.63	6e-05	0.42	0.43	0.43	0.38	0.0007
M27	0.97	0.97	0.97	0.97	0.19087	0.85	0.85	0.85	0.81	5.5e-07	0.64	0.64	0.63	0.59	0.00051	0.4	0.39	0.39	0.34	9.5e-07
M28	0.99	0.99	0.99	0.98	0.00443	0.95	0.94	0.92	0.93	3.8e-06	0.79	0.79	0.77	0.75	9e-05	0.49	0.49	0.46	0.44	3e-05
M29	0.99	0.98	0.98	0.98	0.00046	0.89	0.88	0.89	0.83	2.7e-09	0.64	0.61	0.64	0.54	4.1e-13	0.38	0.4	0.41	0.31	4.4e-14
M30	0.96	0.96	0.96	0.94	1.6e-06	0.79	0.8	0.79	0.75	1.7e-07	0.54	0.55	0.55	0.47	2.5e-11	0.37	0.36	0.36	0.3	1.3e-09