Skip to content

Detailed Results of DDA

Jump to bottom
Huayao edited this page Jul 30, 2019 · 2 revisions

Table 1. Sizes of constrained covering arrays generated by different constraint handlers

Verify Solver Replace P-value
Apache 38.43 38.33 37.13 1.9e-07
Bugzilla 20.13 20.37 19 4.6e-07
GCC 24.93 24.9 23.2 7.2e-13
SPIN-S 25.8 25.8 27.4 2.5e-09
SPIN-V 41.83 41.73 42.8 0.00229
M1 47.93 48.1 47.67 0.30377
M2 36.2 35.7 33.73 1.5e-13
M3 19.77 19.97 19.1 0.00111
M4 26.27 25.97 24.6 2.7e-07
M5 57 56.73 56.97 0.71088
M6 30.5 30.53 27.6 <2e-16
M7 12.43 12.43 11.07 4.7e-13
M8 49.13 49.77 49.23 0.11456
M9 23.07 23.47 21.07 3.1e-11
M10 53.37 53.5 53.27 0.73219
M11 51.93 51.43 51.67 0.26142
M12 47.33 47.57 45.27 1.0e-10
M13 41.83 41.8 40.23 4.2e-06
M14 45.03 44.5 43.7 0.00032
M15 36.47 36.23 36.17 0.62056
M16 29.73 30.13 27 <2e-16
M17 46.73 46.67 45.5 0.00035
M18 51.5 51.3 50.87 0.11087
M19 56.77 56.63 57.9 0.00003
M20 63.37 63.2 63.43 0.71751
M21 43.97 43.87 43.2 0.08327
M22 38.63 38.6 36.27 5.2e-10
M23 15.23 15.1 13.87 2.8e-09
M24 52.27 52.13 51.97 0.70830
M25 58.57 58.57 58.57 1.00000
M26 38.53 38.13 37.67 0.00937
M27 40.2 40 39.3 0.00968
M28 61.2 61.03 60.43 0.01821
M29 33.83 34.23 31.47 9.2e-14
M30 24.17 24.23 22.2 1.0e-12

Table 2. Computational cost (seconds) of the generation algorithm with different constraint handlers

Verify Solver Replace P-value
Apache 3.42 20.95 3.47 <2e-16
Bugzilla 0.16 1.64 0.16 <2e-16
GCC 2.96 16.33 6.06 <2e-16
SPIN-S 0.03 0.61 0.03 <2e-16
SPIN-V 0.63 4.43 1.05 <2e-16
M1 1.72 11.59 3.38 <2e-16
M2 1.07 7.57 1.83 <2e-16
M3 0.05 0.72 0.05 <2e-16
M4 0.28 2.64 0.36 <2e-16
M5 6.73 34.42 21.39 <2e-16
M6 0.52 4.82 0.71 <2e-16
M7 0.03 0.46 0.03 <2e-16
M8 2.47 17.35 5.46 <2e-16
M9 3.21 2.32 3.33 <2e-16
M10 4.97 27.39 14.77 <2e-16
M11 1.85 13.44 3.78 <2e-16
M12 3.31 20.12 6.63 <2e-16
M13 2.3 15.12 4.2 <2e-16
M14 1.22 9.45 1.58 <2e-16
M15 0.5 3.8 0.84 <2e-16
M16 0.68 5.21 0.93 <2e-16
M17 3.28 17.91 8.95 <2e-16
M18 3.51 23.56 6.72 <2e-16
M19 10.79 47.4 50.61 <2e-16
M20 6.3 37.64 16.05 <2e-16
M21 1 8.26 1.52 <2e-16
M22 0.75 6.51 1.07 <2e-16
M23 0.04 0.6 0.04 <2e-16
M24 2.62 16.96 5.23 <2e-16
M25 4.05 24.04 9.79 <2e-16
M26 1.25 8.39 2.5 <2e-16
M27 0.57 4.73 0.87 <2e-16
M28 10 48.2 40.57 <2e-16
M29 2.25 14.12 4.81 <2e-16
M30 0.46 4.1 0.64 <2e-16

Table 3. Proportions of failures detected by covering arrays generated by different constraint handlers

k = 3 k = 4 k = 5 k = 6
Verify Solver Replace P-value Verify Solver Replace P-value Verify Solver Replace P-value Verify Solver Replace P-value
Apache 0.97 0.97 0.98 0.3657 0.87 0.85 0.86 0.08811 0.61 0.62 0.61 0.8089 0.39 0.38 0.37 0.39724
Bugzilla 0.93 0.94 0.94 0.81583 0.7 0.72 0.69 0.03357 0.43 0.44 0.4 0.00652 0.23 0.24 0.21 0.00718
GCC 0.97 0.96 0.95 0.05651 0.78 0.78 0.76 0.13854 0.52 0.54 0.49 0.00241 0.31 0.3 0.29 0.4874
SPIN-S 0.88 0.86 0.87 0.00728 0.56 0.55 0.58 0.03628 0.3 0.31 0.33 0.0297 0.16 0.17 0.17 0.97313
SPIN-V 0.94 0.93 0.93 0.24813 0.71 0.73 0.73 0.08155 0.49 0.5 0.5 0.64796 0.24 0.25 0.26 0.0859
M1 0.98 0.98 0.98 0.1712 0.85 0.85 0.84 0.5893 0.63 0.61 0.63 0.42649 0.41 0.41 0.41 0.96696
M2 0.97 0.97 0.97 0.54059 0.83 0.84 0.81 0.00475 0.59 0.57 0.57 0.2734 0.36 0.35 0.35 0.58306
M3 0.89 0.89 0.89 0.93739 0.7 0.7 0.68 0.42904 0.43 0.42 0.41 0.25461 0.25 0.24 0.25 0.83107
M4 0.94 0.94 0.93 0.39447 0.77 0.77 0.75 0.05058 0.5 0.52 0.49 0.15633 0.28 0.29 0.29 0.99593
M5 0.99 0.99 0.98 0.34323 0.92 0.91 0.92 0.4135 0.7 0.7 0.69 0.46133 0.47 0.46 0.46 0.55303
M6 0.98 0.98 0.96 0.00123 0.85 0.86 0.85 0.59955 0.65 0.62 0.6 0.00039 0.42 0.43 0.39 0.00097
M7 0.89 0.89 0.85 9.2e-07 0.68 0.67 0.65 0.06598 0.4 0.4 0.37 0.13411 0.23 0.22 0.21 0.12837
M8 0.99 0.99 0.98 0.14442 0.89 0.89 0.89 0.96434 0.66 0.66 0.66 0.70521 0.42 0.44 0.43 0.47137
M9 0.95 0.96 0.94 0.00291 0.79 0.79 0.76 0.01406 0.61 0.61 0.6 0.6096 0.39 0.38 0.36 0.13931
M10 0.98 0.98 0.98 0.73881 0.9 0.9 0.88 0.02203 0.68 0.69 0.68 0.30569 0.45 0.43 0.42 0.14338
M11 0.98 0.97 0.98 0.02815 0.85 0.87 0.86 0.05488 0.65 0.66 0.65 0.4079 0.43 0.41 0.4 0.12996
M12 0.98 0.99 0.98 0.01186 0.88 0.89 0.87 0.01451 0.66 0.67 0.64 0.03517 0.48 0.47 0.45 0.05226
M13 0.99 0.99 0.98 0.12773 0.88 0.88 0.87 0.4392 0.66 0.65 0.64 0.46131 0.41 0.41 0.4 0.58406
M14 0.98 0.97 0.97 0.34184 0.88 0.87 0.86 0.25621 0.6 0.61 0.6 0.89678 0.39 0.41 0.36 0.00028
M15 0.96 0.95 0.96 0.81867 0.79 0.8 0.8 0.71279 0.55 0.54 0.55 0.5858 0.34 0.35 0.35 0.96522
M16 0.97 0.97 0.97 0.33407 0.85 0.85 0.83 0.22844 0.68 0.66 0.64 0.01071 0.42 0.44 0.4 0.00102
M17 0.99 0.99 0.99 0.30334 0.89 0.88 0.89 0.96963 0.71 0.7 0.68 0.01825 0.45 0.44 0.43 0.13004
M18 0.98 0.99 0.99 0.44376 0.87 0.87 0.86 0.45141 0.67 0.67 0.66 0.70374 0.4 0.4 0.38 0.10813
M19 0.99 0.99 0.99 0.91505 0.91 0.91 0.92 0.3091 0.69 0.68 0.7 0.04439 0.47 0.49 0.49 0.08102
M20 0.99 0.99 0.99 0.27974 0.9 0.9 0.9 0.85369 0.72 0.7 0.71 0.0825 0.5 0.49 0.52 0.17005
M21 0.99 0.99 0.98 0.0252 0.91 0.9 0.9 0.80335 0.72 0.71 0.7 0.435 0.49 0.5 0.5 0.52472
M22 0.98 0.97 0.97 0.76508 0.87 0.88 0.86 0.02016 0.67 0.65 0.64 0.03977 0.41 0.4 0.4 0.7498
M23 0.89 0.89 0.86 0.00142 0.64 0.65 0.62 0.08083 0.41 0.41 0.38 0.02824 0.22 0.22 0.22 0.97084
M24 0.99 0.99 0.99 0.15719 0.9 0.9 0.9 0.98813 0.68 0.69 0.69 0.5761 0.45 0.47 0.45 0.17044
M25 0.99 0.99 0.99 0.67522 0.86 0.87 0.86 0.18569 0.65 0.67 0.67 0.23323 0.43 0.43 0.42 0.68498
M26 0.97 0.98 0.98 0.33413 0.86 0.84 0.85 0.20539 0.63 0.63 0.62 0.67508 0.38 0.38 0.4 0.18462
M27 0.96 0.96 0.97 0.13434 0.81 0.81 0.81 0.64779 0.59 0.58 0.6 0.25288 0.33 0.34 0.33 0.45539
M28 0.98 0.99 0.98 0.37891 0.91 0.91 0.91 0.81474 0.75 0.73 0.74 0.24645 0.43 0.44 0.43 0.44494
M29 0.98 0.98 0.98 0.98594 0.84 0.85 0.82 0.04471 0.58 0.59 0.56 0.04341 0.36 0.35 0.33 0.05927
M30 0.95 0.95 0.94 0.0135 0.77 0.78 0.73 0.0001 0.51 0.51 0.48 0.02009 0.32 0.34 0.32 0.18586

Implementations

Results

Clone this wiki locally