Doc unit testing

Unit testing in Rust

Rust has built in support for simple unit testing. Functions can be marked as unit tests using the 'test' attribute.

#[test]
fn return_none_if_empty() {
   ... test code ...
}

A test function's signature must have no arguments and no return value. To run the tests in a crate, it must be compiled with the '--test' flag: rustc myprogram.rs --test -o myprogram-tests. Running the resulting executable will run all the tests in the crate. A test is considered successful if its function returns; if the task running the test fails, through a call to fail!, a failed check or assert, or some other means, then the test fails.

When compiling a crate with the '--test' flag '--cfg test' is also implied, so that tests can be conditionally compiled.

#[cfg(test)]
mod tests {
  #[test]
  fn return_none_if_empty() {
    ... test code ...
  }
}

Additionally #[test] items behave as if they also have the #[cfg(test)] attribute, and will not be compiled when the --test flag is not used.

Tests that should not be run can be annotated with the 'ignore' attribute. The existence of these tests will be noted in the test runner output, but the test will not be run. Tests can also be ignored by configuration so, for example, to ignore a test on windows you can write #[ignore(cfg(target_os = "win32"))].

Tests that are intended to fail can be annotated with the 'should_fail' attribute. The test will be run, and if it causes its task to fail then the test will be counted as successful; otherwise it will be counted as a failure. For example:

#[test]
#[should_fail]
fn test_out_of_bounds_failure() {
  let v: [int] = [];
  v[0];
}

A test runner built with the '--test' flag supports a limited set of arguments to control which tests are run: the first free argument passed to a test runner specifies a filter used to narrow down the set of tests being run; the '--ignored' flag tells the test runner to run only tests with the 'ignore' attribute.

Parallelism

By default, tests are run in parallel, which can make interpreting failure output difficult. In these cases you can set the RUST_THREADS environment variable to 1 to make the tests run sequentially.

Benchmarking

The test runner also understands a simple form of benchmark execution. Benchmark functions are marked with the #[bench] attribute, rather than #[test], and have a different form and meaning. They are not run by default.

The type signature of a benchmark function differs from a unit test: it takes a mutable reference to type test::BenchHarness. Inside the benchmark function, any time-variable or "setup" code should execute first, followed by a call to iter on the benchmark harness, passing a closure that contains the portion of the benchmark you wish to actually measure the per-iteration speed of. For example:

extern mod std;

#[bench]
fn bench_sum_1024_ints(b: &mut std::test::BenchHarness) {
    let v = vec::from_fn(1024, |n| n);
    do b.iter {
        v.foldl(|a, b| a + b);
    }
}

The benchmark runner will calibrate measurement of the benchmark function to run the iter block "enough" times to get a reliable measure of the per-iteration speed. In other words, you do not have to (and should not) put any logic for calibration or "picking a good benchmark size" into your test: so long as it represents a loop-body that's "large enough to measure with a high-precision clock" -- more than a few microseconds -- it should be fine. Let the benchmark runner calibrate it. If the benchmark runner doesn't do a good job -- too much noise or faulty statistical reasoning -- that's a bug in its logic to correct; benchmarks that are too elaborate to optimize successfully or too slow to run regularly are less useful in the long run.

To run benchmarks, pass the --bench flag to the compiled test-runner. Benchmarks are compiled-in but not executed by default.

Examples

Typical test run

> mytests

running 30 tests
running driver::tests::mytest1 ... ok
running driver::tests::mytest2 ... ignored
... snip ...
running driver::tests::mytest30 ... ok

result: ok. 28 passed; 0 failed; 2 ignored

Test run with failures

> mytests

running 30 tests
running driver::tests::mytest1 ... ok
running driver::tests::mytest2 ... ignored
... snip ...
running driver::tests::mytest30 ... FAILED

result: FAILED. 27 passed; 1 failed; 2 ignored

Running ignored tests

> mytests --ignored

running 2 tests
running driver::tests::mytest2 ... failed
running driver::tests::mytest10 ... ok

result: FAILED. 1 passed; 1 failed; 0 ignored

Running a subset of tests

> mytests mytest1

running 11 tests
running driver::tests::mytest1 ... ok
running driver::tests::mytest10 ... ignored
... snip ...
running driver::tests::mytest19 ... ok

result: ok. 11 passed; 0 failed; 1 ignored

Running benchmarks

> mytests --bench

running 1 test
test bench_sum_1024_ints ... bench: 65888 ns/iter (+/- 309)

result: ok. 0 passed; 0 failed; 0 ignored