-
Notifications
You must be signed in to change notification settings - Fork 1
Home
The UnitTestToolbox implements a MATLAB class, @UnitTest, which enables the user to validate
- the runtime execution of a set of Matlab scripts and
- the results generated by these scripts.
The @UnitTest class can be useful to establish that the specified MATLAB scripts run without runtime errors and that the data they produce across different machines/architectures are consistent with a ground truth data set. The class is also useful to identify instances in which updates to the core functionality of a MATLAB project produce results that differ from those produced by earlier versions.
The toolbox is used heavily by ISETValidate, which oversees the code in the ISETCam, ISETBio, and ISET3d repositories. Please see ieValidate() for an example of how we use this toolbox for the ISET repositories.
UnitTest-based validations compare data generated by a set of Matlab scripts to those contained in corresponding ground truth data sets. There are two main types of validations:
- 'FAST', based on the hash signatures of the truncated validation data
- 'FULL', based on full numerical comparison of the validation data
The first step in adapting the @UnitTest toolbox for a particular project is to customize this configuration script, found in the UnitTestTools folder. The various customization options are specified as fields in a Matlab structure. The following listing shows the settings for a project called 'theProject'. Adapt to your needs.
% Specify project-specific preferences
p = struct(...
'projectName', 'theProject', ... % The project's name (also the preferences group name)
'validationRootDir', '/path/ToTopLevelDirectory', ... % Directory location where the 'scripts' subdirectory resides.
'alternateFastDataDir', '', ... % Alternate FAST (hash) data directory location. Specify '' to use the default location, i.e., $validationRootDir/data/fast
'alternateFullDataDir', '', ... % fullfile(filesep,'Users', 'Shared', 'Dropbox', 'theProjectFullValidationData'), ... % Alternate FULL data directory location. Specify '' to use the default location, i.e., $validationRootDir/data/full
'clonedWikiLocation', fullfile(filesep,'Users', 'Shared', 'Matlab', 'Toolboxes', 'theProject_Wiki', 'theProject.wiki'), ... % Local path to the directory where the wiki is cloned. Only relevant for publishing tutorials.
'clonedGhPagesLocation', fullfile(filesep,'Users', 'Shared', 'Matlab', 'Toolboxes', 'theProject_GhPages', 'theProject'), ... % Local path to the directory where the gh-pages repository is cloned. Only relevant for publishing tutorials.
'githubRepoURL', 'http://theProject.github.io/theProject', ... % Github URL for the project. This is only used for publishing tutorials.
'generateGroundTruthDataIfNotFound', false ... % Flag indicating whether to generate ground truth if one is not found
);
where /path/ToTopLevelDirectory/scripts/ contains directories with validation scripts.
This script is typically run only once. If one setting is changed the script would need to be rerun so as to update the projects's settings which are stored as a matlab prefs entity.
Important: Non-developers should set generateGroundTruthDataIfNotFound=false. Only the project developers should enable this flag when they need to generate new ground truth data.
To conduct a validation of a set of Matlab scripts you can adapt this executive validation script template, which is also found in the UnitTestDemos folder. In the following, we go through this script section by section to explain how things work.
We begin by specifying the project name, here the 'theProject' project. If a second argument is passed and set to 'reset', all preferences are reset to their default values. If no second argument is passed, the existing preferences for the specified project will be used.
function ExecutiveValidationScriptTemplate
% Use preferences for the 'theProject' project
UnitTest.usePreferencesForProject('theProject', 'reset');
To see the current UnitTest preferences you can type the following in Matlab's command window:
UnitTest.listPrefs
Next, we can change some of these preferences.
%% Run time error behavior
% valid options are: 'rethrowExceptionAndAbort', 'catchExceptionAndContinue'
UnitTest.setPref('onRunTimeErrorBehavior', 'catchExceptionAndContinue');
%% Plot generation
UnitTest.setPref('generatePlots', false);
UnitTest.setPref('closeFigsOnInit', true);
%% Verbosity Level
% valid options are: 'none', min', 'low', 'med', 'high', 'max'
UnitTest.setPref('verbosity', 'med');
%% Numeric tolerance for comparison to ground truth data
UnitTest.setPref('numericTolerance', 500*eps);
%% Whether to plot data that do not agree with the ground truth
UnitTest.setPref('graphMismatchedData', true);
After the preferences are set, we proceed with specifying the set of scripts to validate. We can choose to validate scripts in select directories as follows:
% Get the validation rootDir
rootDir = UnitTest.getPref('validationRootDir');
% List of script directories to validate. Each entry contains a cell array with
% with a validation script directory and an optional struct with
% prefs that override the corresponding isetbioValidation prefs.
% At the moment only the 'generatePlots' pref can be overriden.
vScriptsList = {...
{fullfile(rootDir, 'scripts', 'color')} ...
{fullfile(rootDir, 'scripts', 'cones')} ...
{fullfile(rootDir, 'scripts', 'human')} ...
{fullfile(rootDir, 'scripts', 'codedevscripts'), struct('generatePlots', false) } ...
};
Alternatively, we can choose to validate individual scripts as follows:
% Get rootDir
rootDir = UnitTest.getPref('validationRootDir');
% List of scripts to validate.
% Each entry contains a cell array with
% with a script name and an optional struct with
% prefs that override the corresponding isetbioValidation prefs.
% At the moment only the generatePlots pref can be overriden.
vScriptsList = {...
{fullfile(rootDir, 'scripts', 'color', 'v_stockman2xyz.m')}
{fullfile(rootDir, 'scripts', 'codedevscripts', 'v_skeleton.m'), struct('generatePlots', false) }
};
Finally, we specify what kind of validation to perform. There are four types of validation:
-
RUN_TIME_ERRORS_ONLY, which tests whether the scripts run without any runtime errors, without validating any generated data, -
FAST, which first truncates the generated validation data to 9 decimal digits and subsequently checks whether the hash of the truncated data agree with that of the ground truth, -
FULL, which fully compares the generated validation data against the ground truth and identifies any data whose values deviate from those of the ground truth, -
PUBLISH, which does aFULLvalidation and publishes the executed scripts and their output in the gh-pages repository of the project. The four different validation types can be conducted as follows. Uncomment the one that suits you best.
%% How to validate
% Run a RUN_TIME_ERRORS_ONLY validation session
%UnitTest.runValidationSession(vScriptsList, 'RUN_TIME_ERRORS_ONLY')
% Or run a FAST validation session (comparing SHA-256 hash keys of the data)
%UnitTest.runValidationSession(vScriptsList, 'FAST');
% Or run a FULL validation session (comparing actual data)
%UnitTest.runValidationSession(vScriptsList, 'FULL');
% Or run a PUBLISH validation session (comparing actual data and update github wiki)
%UnitTest.runValidationSession(vScriptsList, 'PUBLISH);
% Or run a validation session without a specified mode. You will be
% promped to select one of the available modes.
%UnitTest.runValidationSession(vScriptsList);
This section describes how to convert matlab scripts into UnitTest object manageable functions. All such functions must contain a wrapper function followed by a validation function.
The purpose of the wrapper function is to call the validation function and to setup a communication channel between the validation function and the managing @UnitTest object. This wrapper function also ensures that the validation script can be run in stand-alone mode, i.e., without being managed by a @UnitTest object.
The validation function contains the original Matlab script that is to be validated with interspersed calls to @UnitTest methods for logging validation data and messages to the managing @UnitTest object.
The following programming structure must be followed:
function varargout = v_computeSomething(varargin)
%
% Single line documentation script, describing the types of computation performed by this script. This information appears in the project's github site when run in 'PUBLISH' mode.
%
varargout = UnitTest.runValidationRun(@ValidationFunction, nargout, varargin);
end
%% Actual validation code
function ValidationFunction(runTimeParams)
...
matlab code and calls to UnitTest methods for logging validation data and messages
...
end
The UnitTest class allows two types of data logging:
- validation data, which are used for the external checks against the ground truth
- extra data, which are additional data, not used during the validation, but nevertheless stored in the ground truth as reference data.
Data logging is achieved by the following calls:
UnitTest.validationData(fieldName, fieldValue);
UnitTest.extraData(fieldName, fieldValue);
The UnitTest class also allows various types of message logging. The following message types are available:
- simple informative messages with no special status,
- messages indicating that a fundamental internal check failed
- messages indicating that an internal check failed
- messages indicating that an internal check passed
Message logging is achieved by the following calls:
UnitTest.validationRecord('SIMPLE_MESSAGE', messageString);
UnitTest.validationRecord('FUNDAMENTAL_CHECK_FAILED', messageString);
UnitTest.validationRecord('FAILED', messageString);
UnitTest.validationRecord('PASSED', messageString);
01: function varargout = v_skeleton(varargin)
02: %
03: % Skeleton script containing the minimally required code. Copy and add your ISETBIO validation code.
04: %
05: % ONE LINE COMMENT ABOVE WILL GET AUTOPUBLISHED AS THE DESCRIPTION OF THIS SCRIPT.
06:
07: varargout = UnitTest.runValidationRun(@ValidationFunction, nargout, varargin);
08: end
09:
10:
11: %% Function implementing the isetbio validation code
12: function ValidationFunction(runTimeParams)
13:
14: %% script validation code
15:
16: %% Some informative text
17: UnitTest.validationRecord('SIMPLE_MESSAGE', 'Validating computation 1.');
18:
19: %% more script validation code
20:
21: %% Internal validations
22: %
23: % Check whether quantity is within tolerance of zero
24: quantityOfInterest = randn(100,1)*0.0000001;
25: tolerance = 0.000001;
26: UnitTest.assertIsZero(quantityOfInterest,'Result',tolerance);
27:
28: % If you want to do a more complicated comparison, you could write
29: % things out more fully along the lines of the commented out code
30: % here:
31: %
32: % if (max(abs(quantityOfInterest) > tolerance))
33: % message = sprintf('Result exceeds specified tolerance (%0.1g). !!!', tolerance);
34: % UnitTest.validationRecord('FAILED', message);
35: % else
36: % message = sprintf('Result is within the specified tolerance (%0.1g).', tolerance);
37: % UnitTest.validationRecord('PASSED', message);
38: % end
39:
40: %% Simple assertion
41: fundamentalCheckPassed = true;
42: UnitTest.assert(fundamentalCheckPassed,'fundamental assertion');
43:
44: % You can also do a customized assert that only prints on failure
45: % as in the commented out code below. This would also allow you to
46: % put a return after the check, to abort trying to go along further.
47: % This method produces a more agressive error message and should be
48: % reserved only for cases where something is very deeply wrong should
49: % the assertion fail.
50: % if (~fundamentalCheckPassed)
51: % UnitTest.validationRecord('FUNDAMENTAL_CHECK_FAILED', 'A fundamental check failed');
52: % end
53:
54:
55: %% Data for external validations
56: dataA = ones(10,20);
57:
58: % Add validation data - these will be contrasted against the
59: % ground truth data with respect to a specified tolerance level
60: % and the result will determine whether the validation passes or fails.
61: UnitTest.validationData('variableNameForDataA', dataA);
62:
63: %% Data to keep just because it would be nice to have
64: dataB = rand(10,30);
65:
66: % Add extra data - these will be contrasted against their stored
67: % counterpants only when the verbosity level is set to 'med' or higher,
68: % and only when running in 'FULL' validation mode.
69: % The validation status does not depend on the status of these comparisons.
70: % This can be useful for storing variables that have a stochastic component.
71: UnitTest.extraData('variableNameForDataB', dataB);
72:
73: %% Plotting
74: if (runTimeParams.generatePlots)
75: figure(7);
76: clf;
77: plot(1:numel(quantityOfInterest ), quantityOfInterest , 'k-');
78: hold on;
79: plot([1 numel(quantityOfInterest )], tolerance*[1 1], 'r-');
80: set(gca, 'YLim', [min(quantityOfInterest ) max([max(quantityOfInterest ) tolerance])*1.1]);
81: drawnow;
82: end
83: end