Add --test-tool=nextest

[sourcefrog]

Fixes #85

• Update manual
• Exercise from cargo-mutants' own CI
• Integration test
• Check docs about passing down options to cargo test also cover nextest

Performance on cargo-mutants

I tested this on cargo-mutants itself locally and in CI, and it works well. However, interestingly, it does currently seem to be significantly slower in sharded full-tree tests than is cargo test (about 8m vs >13m). (Example CI run.)

It's also somewhat slower, although not as much, locally:

; c r --release mutants --shard 1/10 --test-tool=nextest
   Compiling cargo-mutants v23.12.2 (/home/mbp/src/mutants)
    Finished release [optimized] target(s) in 4.49s
     Running `target/release/cargo-mutants mutants --shard 1/10 --test-tool=nextest`
Found 56 mutants to test
ok       Unmutated baseline in 13.9s build + 22.3s test
Auto-set test timeout to 1m 51s
56 mutants tested in 4m 22s: 40 caught, 16 unviable


; c r --release mutants --shard 1/10
    Finished release [optimized] target(s) in 0.08s
     Running `target/release/cargo-mutants mutants --shard 1/10`
Found 56 mutants to test
ok       Unmutated baseline in 13.8s build + 21.7s test
Auto-set test timeout to 1m 48s
56 mutants tested in 3m 46s: 40 caught, 16 unviable

Although it's probably better at stopping early, it may be paying a higher per-test overhead or maybe making less use of parallelism?

Performance on stackslib

I also tested this on a 1% sample of stackslib with

; c r --release mutants --shard 1/100 -d ~/src/stacks-core/ --test-tool nextest
    Finished release [optimized] target(s) in 0.18s
     Running `target/release/cargo-mutants mutants --shard 1/100 -d /home/mbp/src/stacks-core/ --test-tool nextest`
Found 112 mutants to test
ok       Unmutated baseline in 116.1s build + 244.5s test
Auto-set test timeout to 20m 22s
MISSED   src/burnchains/bitcoin/spv.rs:1150:55: replace > with == in SpvClient::get_target in 34.1s build + 256.1s test
MISSED   src/chainstate/burn/db/sortdb.rs:2893:25: replace == with != in SortitionDB::is_db_version_supported_in_epoch in 25.8s build + 283.8s test
3358.435171577s  INFO timeout after 1222.4s, terminating child process...
TIMEOUT  src/net/connection.rs:1334:9: replace NetworkConnection<P>::send_data -> Result<usize, net_error> with Ok(1) in 34.8s build + 1222.4s test
MISSED   src/chainstate/coordinator/comm.rs:105:9: replace SignalBools::receive_signal -> u8 with 0 in 38.0s build + 278.8s test
MISSED   src/main.rs:957:16: replace == with != in main in 24.3s build + 258.5s test
MISSED   src/net/p2p.rs:4235:9: replace PeerNetwork::need_block_or_microblock_stream -> Result<bool, net_error> with Ok(true) in 28.7s build + 267.8s test
MISSED   src/net/p2p.rs:1280:9: replace PeerNetwork::num_peers -> usize with 1 in 26.8s build + 311.8s test
test     src/clarity_vm/database/mod.rs:174:9: replace <impl HeadersDB for ChainstateTx<'a>>::get_burnchain_tokens_spent_for_block -> Option<u128> with Some(1) ... 233.0s
MISSED   src/net/http.rs:1730:9: replace HttpRequestType::get_proof_query -> bool with true in 33.5s build + 291.0s test
MISSED   src/util_lib/db.rs:656:20: replace > with == in tx_busy_handler in 35.5s build + 290.3s test
test     src/chainstate/stacks/db/unconfirmed.rs:443:9: replace UnconfirmedState::num_mined_txs -> usize with 0 s
└                PASS [  28.317s]                     blockstack-core clarity_vm::tests::costs::epoch_21_test_alt
47/112 mutants tested, 8 MISSED, 1 timeout, 24 caught, 14 unviable, 1h 49m 40s elapsed, about 2h 32m remaining

It looks like some of those tests are still pretty slow, but it is making progress.

[sunshowers]

Looks pretty good!

[ASuciuX]

Performance on stackslib
I also tested this on a 1% sample of stackslib with

; c r --release mutants --shard 1/100 -d ~/src/stacks-core/ --test-tool nextest

Hey @sourcefrog , thanks for updating cargo mutants with nextest!
In the above example, the mutants that you specified are run from the whole stacks-core workspace, not specifically from stackslib.

I've tested it on stackslib with mutants that are caught and the difference is truly significant ( excepting unmutated build and test, from 1 hour and 10 minutes to 6 minutes).

cargo test

; command containing the regex of the 4 mutants tested
; test-threads=1 and bitcoind_test=1 are required for stackslib in order to work properly with bitcoin
; BITCOIND_TEST=1 RUST_BACKTRACE=full cargo mutants -F "(stackslib/src/chainstate/nakamoto/mod.rs:2595:9: replace NakamotoChainState::set_aggregate_public_key with \(\))|(stackslib/src/chainstate/nakamoto/mod.rs:3059:9: replace NakamotoChainState::aggregate_public_key_bootcode with \(\))|(stackslib/src/net/relay.rs:661:9: replace Relayer::process_new_nakamoto_block -> Result<bool, chainstate_error> with Ok\(true\))|(stackslib/src/net/relay.rs:661:9: replace Relayer::process_new_nakamoto_block -> Result<bool, chainstate_error> with Ok\(false\))" -- --all-targets -- --test-threads 1

Found 4 mutants to test
ok       Unmutated baseline in 131.2s build + 1052.1s test
Auto-set test timeout to 1h 27m 40s
4 mutants tested in 1h 31m 41s: 4 caught

cargo nextest

; BITCOIND_TEST=1 RUST_BACKTRACE=full cargo mutants -F "(stackslib/src/chainstate/nakamoto/mod.rs:2595:9: replace NakamotoChainState::set_aggregate_public_key with \(\))|(stackslib/src/chainstate/nakamoto/mod.rs:3059:9: replace NakamotoChainState::aggregate_public_key_bootcode with \(\))|(stackslib/src/net/relay.rs:661:9: replace Relayer::process_new_nakamoto_block -> Result<bool, chainstate_error> with Ok\(true\))|(stackslib/src/net/relay.rs:661:9: replace Relayer::process_new_nakamoto_block -> Result<bool, chainstate_error> with Ok\(false\))" --test-tool=nextest -- --fail-fast --test-threads 1

Found 4 mutants to test
ok       Unmutated baseline in 130.8s build + 1054.7s test
Auto-set test timeout to 1h 27m 53s
4 mutants tested in 25m 22s: 4 caught

The baseline tests ran in ~20 minutes, meaning it took less than 6 minutes to run the mutants with nextest's fail-fast flag enabled, compared to cargo test's 1h 10m.

Documentation related as well, the -- --test-threads arg from cargo test does not apply to cargo nextest, which doesn't require the additional --, leaving it just as --test-threads.

[sunshowers]

Wow that's incredible @ASuciuX!

[sourcefrog]

That's outstanding, thanks for letting us know!

I will do the couple of small items in the head of the PR including checking the docs about passing options to cargo test/nextest, and then push a release.

I don't think you need to set --fail-fast explicitly for nextest as that's the default.

Regarding test-threads, from previous conversations I think you are setting this because you have some non-hermetic tests that cannot be run in parallel. nextest actually has some nice specific features for this which once set up properly might give you more precise control.

[ASuciuX]

You are right @sourcefrog , I've tried without --fail-fast and I had the same results.

Interesting features, thank you for sharing, I will take a look into those.