Improved docs on CI sharding (#302)

book/src/shards.md

@@ -1,6 +1,6 @@
 # Sharding
 
-In addition to [running multiple jobs locally](parallelism.md), cargo-mutants can also run jobs on multiple machines, to get an overall job faster.
+In addition to [running multiple jobs locally](parallelism.md), cargo-mutants can also run jobs on multiple machines, to get an overall result faster by using more CPU cores.
 
 Each job tests a subset of mutants, selected by a shard. Shards are described as `k/n`, where `n` is the number of shards and `k` is the index of the shard, from 0 to `n-1`.
 
@@ -11,7 +11,7 @@ If any shard fails then that would indicate that some mutants were missed, or th
 ## Consistency across shards
 
 **CAUTION:**
-All shards must be run with the same arguments, and the same sharding `k`, or the results will be meaningless, as they won't agree on how to divide the work.
+All shards must be run with the same arguments, and the same sharding denominator `n`, or the results will be meaningless, as they won't agree on how to divide the work.
 
 Sharding can be combined with filters or shuffling, as long as the filters are set consistently in all shards. Sharding can also combine with `--in-diff`, again as long as all shards see the same diff.
 
@@ -22,39 +22,22 @@ Your CI system or other tooling is responsible for launching multiple shards, an
 For example, in GitHub Actions, you could use a matrix job to run multiple shards:
 
 ```yaml
-  cargo-mutants:
-    runs-on: ubuntu-latest
-    # needs: [build, incremental-mutants]
-    strategy:
-      matrix:
-        shard: [0, 1, 2, 3, 4, 5, 6, 7]
-    steps:
-      - uses: actions/checkout@v4
-      - uses: dtolnay/rust-toolchain@master
-        with:
-          toolchain: beta
-      - uses: Swatinem/rust-cache@v2
-      - run: cargo install cargo-mutants
-      - name: Mutants
-        run: |
-          cargo mutants --no-shuffle -vV --shard ${{ matrix.shard }}/8
-      - name: Archive mutants.out
-        uses: actions/upload-artifact@v4
-        if: always()
-        with:
-          name: mutants.out
-          path: mutants.out
+{{#include ../../examples/workflows/sharded.yml}}
 ```
 
 Note that the number of shards is set to match the `/8` in the `--shard` argument.
 
+## Skipping the baseline
+
+[Sharding works with `--baseline=skip`](baseline.md), to avoid the cost of running the baseline on every shard. But, if you do this, then you must ensure that the tests suite is passing in the baseline, for example by checking it in a previous CI step.
+
 ## Performance of sharding
 
 Each mutant does some constant upfront work:
 
 * Any CI setup including starting the machine, getting a checkout, installing a Rust toolchain, and installing cargo-mutants
 * An initial clean build of the code under test
-* A baseline run of the unmutated code
+* A baseline run of the unmutated code (unless this is skipped)
 
 Then, for each mutant in its shard, it does an incremental build and runs all the tests.
 
@@ -63,30 +46,30 @@ Each shard runs the same number of mutants, +/-1. Typically this will mean they
 A rough model for the overall execution time for all of the shards, allowing for this work occurring in parallel, is
 
 ```raw
-SHARD_STARTUP + (CLEAN_BUILD + TEST) + (N_MUTANTS/K) * (INCREMENTAL_BUILD + TEST)
+SHARD_STARTUP + (CLEAN_BUILD + TEST) + (N_TOTAL_MUTANTS / N_SHARDS) * (INCREMENTAL_BUILD + TEST)
 ```
 
 The total cost in CPU seconds can be modelled as:
 
 ```raw
-K * (SHARD_STARTUP + CLEAN_BUILD + TEST) + N_MUTANTS * (INCREMENTAL_BUILD + TEST)
+N_SHARDS * (SHARD_STARTUP + CLEAN_BUILD + TEST) + N_MUTANTS * (INCREMENTAL_BUILD + TEST)
 ```
 
-As a result, at very large `k` the cost of the initial setup work will dominate, but overall time to solution will be minimized.
+As a result, if you use many shards the cost of the initial build will dominate, and the overall time will converge towards the time for a clean build, a baseline test, and the test of one mutant.
 
 ## Choosing a number of shards
 
-Because there's some constant overhead for every shard there will be diminishing returns and increasing ineffiency if you use too many shards. (In the extreme cases where there are more shards than mutants, some of them will do the setup work, then find they have nothing to do and immediately exit.)
+Because there's some constant overhead for every shard there will be diminishing returns and increasing ineffiency if you use too many shards. (In the extreme cases where there are more shards than mutants, some of them will find they have nothing to do and immediately exit.)
 
-As a rule of thumb, you should probably choose `k` such that each worker runs at least 10 mutants, and possibly much more. 8 to 32 shards might be a good place to start.
+As a rule of thumb, you should probably choose `n` such that each worker runs at least 10 mutants, and possibly much more. 8 to 32 shards might be a good place to start.
 
 The optimal setting probably depends on how long your tree takes to build from zero and incrementally, how long the tests take to run, and the performance of your CI system.
 
 If your CI system offers a choice of VM sizes you might experiment with using smaller or larger VMs and more or less shards: the optimal setting probably also depends on your tree's ability to exploit larger machines.
 
 You should also think about cost and capacity constraints in your CI system, and the risk of starving out other users.
 
-cargo-mutants has no internal scaling constraints to prevent you from setting `k` very large, if cost, efficiency and CI capacity are not a concern.
+cargo-mutants has no internal scaling constraints to prevent you from setting `n` very large, if cost, efficiency and CI capacity are not a concern.
 
 ## Sampling mutants
 

examples/workflows/sharded.yml

@@ -0,0 +1,81 @@
+# Example of using a GitHub Actions matrix to shard the mutants run into 8 parts.
+
+# See https://github.com/sourcefrog/cargo-mutants/blob/main/.github/workflows/tests.yml for a full example.
+
+# Only run this on PRs or main branch commits that could affect the results,
+# so we don't waste time on doc-only changes. Adjust these paths and branch names
+# to suit your project.
+on:
+  pull_request:
+    paths:
+      - ".cargo/*.toml"
+      - ".github/workflows/tests.yml"
+      - "Cargo.*"
+      - "mutants_attrs/**"
+      - "src/**"
+      - "testdata/**"
+      - "tests/**"
+  push:
+    branches:
+      - main
+    # Actions doesn't support YAML references, so it's repeated here
+    paths:
+      - ".cargo/*.toml"
+      - ".github/workflows/tests.yml"
+      - "Cargo.*"
+      - "mutants_attrs/**"
+      - "src/**"
+      - "testdata/**"
+      - "tests/**"
+
+jobs:
+  # Before testing mutants, run the build and tests on all platforms.
+  # You probably already have CI configuration like this, so don't duplicate it,
+  # merge cargo-mutants into your existing workflow.
+  test:
+    strategy:
+      matrix:
+        os: [macOS-latest, ubuntu-latest, windows-latest]
+        version: [stable, nightly]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v4
+      - uses: dtolnay/rust-toolchain@master
+        with:
+          toolchain: ${{ matrix.version }}
+          components: rustfmt
+      - uses: swatinem/rust-cache@v2
+      - name: rustfmt
+        run: cargo fmt --all -- --check
+      - name: Build
+        run: cargo build --all-targets
+      - name: Test
+        run: cargo test --workspace
+  cargo-mutants:
+    runs-on: ubuntu-latest
+    # Often you'll want to only run this after the build is known to pass its basic tests,
+    # to avoid wasting time, and to allow using --baseline=skip.
+    needs: [test]
+    strategy:
+      fail-fast: false # Collect all mutants even if some are missed
+      matrix:
+        shard: [0, 1, 2, 3, 4, 5, 6, 7]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: dtolnay/rust-toolchain@master
+      - uses: Swatinem/rust-cache@v2
+      - uses: taiki-e/install-action@v2
+        name: Install cargo-mutants using install-action
+        with:
+          tool: cargo-mutants
+      # Set an appropriate timeout for your tree here.
+      # The denominator of the shard count must be the number of shards.
+      - name: Mutants
+        run: |
+          cargo mutants --no-shuffle -vV --shard ${{ matrix.shard }}/8 --baseline=skip --timeout 300 --in-place
+      - name: Archive mutants.out
+        uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          path: mutants.out
+          name: mutants-shard${{matrix.shard}}.out