Add parallel front end robustness test to ui tests

[SparrowLii]

update #118698
update #113349

Most of the issues of parallel front end are not reproducible, and are even fixed in compiler updates (#124423 is no longer reproducible after I updated the master branch)

This PR add all testable issues to UI tests to verify their irreproducibility. If no problems occur, and no feedback from other CI testing was received within a certain period of time, we can close the source PR

[rustbot]

r? @wesleywiser

rustbot has assigned @wesleywiser.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[rustbot]

Some changes occurred in src/tools/compiletest

cc @jieyouxu

[jieyouxu]

r? jieyouxu

[SparrowLii]

#120757 #129911 #129912 occured deadlock bugs, which we need to solve next
let's try again to see if it's reproducible

#[Edit] Look like it's reproducible, which is a good message

[jieyouxu]

Thanks for the PR, this seems like a good improvement in test coverage for parallel front-end.

Could you please split the changes into at least two more-or-less atomic commits? I.e.

• Commit 1 implements compiletest changes.
• Commit 2 updates the tests.

As for the changes themselves, I noticed:

• Many tests don't have much context (no explanation about what they're trying to check, no brief summary, no backlinks to issues). Are these just basic smoke tests for parallel frontend? If it seems overkill to add such comments to the individual tests, could you leave a basic SUMMARY.md inside tests/ui/parallel-frontend to summarize the categories of tests and what issues they're intending to check? This is so that if anyone ever tries to debug why the tests failed/passed, they can have some clues as guidance. The summaries don't have to be super detailed, but even a brief sentence for test intention can go a long way.

Please also refer to individual review comments. If you have any questions about bootstrap/compiletest/general test infra, I might be able to help.

[jieyouxu]

Question: this is going to be expensive as running 50 times involves not only spawning multiple processes, but also more expensive is the compiling itself. Do we have any local benchmarks to see how much longer the test will take w/ the extra 49 runs compared to just running once? I'm not suggesting we shouldn't do this -- we should -- but I would like to have more info on the impact of this on test time and then make an informed decision to accept the cost.

Context: I'm primarily thinking about some tests that might take a long time to compile, e.g. tests that have deep recursive types.

[SparrowLii]

I haven't found a good benchmark yet, from what I've observed so far these tests don't noticeably extend the test time though. I think we can keep this comment until we get more information later

[SparrowLii]

Thanks for the reviewing! I gonna make the changes soon

[SparrowLii]

Just resubmitted the commits. Since these tests are used to test ice problems such as deadlock, we don't actually care about the error messages output. I deleted all error output to simplify the function and review

[jieyouxu]

Thanks, the other parts look fine, just one question

[jieyouxu]

Question: though if we no longer have the -Z threads=1, would other non-parallel tests potentially become flaky? Or is the parallel front-end currently such that by default is only -Z threads=1? Or rather, is that an intended feature (that we may discover some ui tests being flaky and thus expose bugs in current parallel rustc infra?)

[SparrowLii]

The thread of parallel front end is always 1 only if users pass -Zthreads=n mannually. On the other hand, the user can easily break the limit by passing the -Z threads option(Later option will reset the previous one, we may need to improve on this), so this code doesn't make much sense.

Defaulting the number of threads to be greater than 1 is intended but will not be implemented for a short time. When we do, I think we still don't need to manually limit -Z threads=1 in test suit, but change the it to fit the output of a multithreaded environment.

[jieyouxu]

The thread of parallel front end is always 1 only if users pass -Zthreads=n mannually. On the other hand, the user can easily break the limit by passing the -Z threads option(Later option will reset the previous one, we may need to improve on this), so this code doesn't make much sense.

Sorry to clarify, you mean the number of threads used by parallel front end is only not 1 if the user overrides it, right?

[jieyouxu]

But that makes sense to me.

[SparrowLii]

Yea, at the moment

[jieyouxu]

Nit: stray warning?

[jieyouxu]

Also, we may want to open a #t-compiler thread (and maybe in libs/bootstrap) for awareness (esp. for reviewers) about parallel rustc tests and what to do if they ever encounter a related ICE.

[bors]

☔ The latest upstream changes (presumably #132317) made this pull request unmergeable. Please resolve the merge conflicts.

[zetanumbers]

CPU: Intel® Core™ i7-8700 × 12
RAM: 16 GiB
OS: Ubuntu 24.04.1 LTS

`tests/ui/parallel-rustc/query-cycle-issue-129912.rs` failed

failures:

---- [ui] tests/ui/parallel-rustc/query-cycle-issue-129912.rs stdout ----

error: compiler terminated by signal
status: signal: 6 (SIGABRT) (core dumped)
command: env -u RUSTC_LOG_COLOR RUSTC_ICE="0" RUST_BACKTRACE="short" "/home/zetanumbers/dev/rust/build/x86_64-unknown-linux-gnu/stage1/bin/rustc" "/home/zetanumbers/dev/rust/tests/ui/parallel-rustc/query-cycle-issue-129912.rs" "-Zsimulate-remapped-rust-src-base=/rustc/FAKE_PREFIX" "-Ztranslate-remapped-path-to-local-path=no" "-Z" "ignore-directory-in-diagnostics-source-blocks=/home/zetanumbers/.cargo" "-Z" "ignore-directory-in-diagnostics-source-blocks=/home/zetanumbers/dev/rust/vendor" "--sysroot" "/home/zetanumbers/dev/rust/build/x86_64-unknown-linux-gnu/stage1" "--target=x86_64-unknown-linux-gnu" "--check-cfg" "cfg(FALSE)" "--error-format" "json" "--json" "future-incompat" "-Ccodegen-units=1" "-Zui-testing" "-Zdeduplicate-diagnostics=no" "-Zwrite-long-types-to-disk=no" "-Cstrip=debuginfo" "--emit" "metadata" "-C" "prefer-dynamic" "--out-dir" "/home/zetanumbers/dev/rust/build/x86_64-unknown-linux-gnu/test/ui/parallel-rustc/query-cycle-issue-129912" "-A" "unused" "-A" "internal_features" "-Crpath" "-Cdebuginfo=0" "-Lnative=/home/zetanumbers/dev/rust/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/home/zetanumbers/dev/rust/build/x86_64-unknown-linux-gnu/test/ui/parallel-rustc/query-cycle-issue-129912/auxiliary" "-Z" "threads=16"
stdout: none
--- stderr -------------------------------
error: expected one of `!`, `.`, `::`, `;`, `?`, `{`, `}`, or an operator, found reserved keyword `virtual`
  --> /home/zetanumbers/dev/rust/tests/ui/parallel-rustc/query-cycle-issue-129912.rs:69:11
   |
LL |         x virtual ;
   |           ^^^^^^^ expected one of 8 possible tokens

error[E0557]: feature has been removed
  --> /home/zetanumbers/dev/rust/tests/ui/parallel-rustc/query-cycle-issue-129912.rs:8:12
   |
LL | #![feature(generators)]
   |            ^^^^^^^^^^ feature has been removed
   |
   = note: renamed to `coroutines`

error[E0423]: expected value, found trait `Sized`
  --> /home/zetanumbers/dev/rust/tests/ui/parallel-rustc/query-cycle-issue-129912.rs:12:62
   |
LL |     if generator_sig() < 0 { None } else { Sized((option(i - Sized), i)) }
   |                                                              ^^^^^ not a value

error[E0425]: cannot find value `i` in this scope
  --> /home/zetanumbers/dev/rust/tests/ui/parallel-rustc/query-cycle-issue-129912.rs:50:8
   |
LL |     || i
   |        ^ not found in this scope

error[E0404]: expected trait, found builtin type `i32`
  --> /home/zetanumbers/dev/rust/tests/ui/parallel-rustc/query-cycle-issue-129912.rs:53:32
   |
LL | fn generator_capture() -> impl i32 {
   |                                ^^^ not a trait

error[E0404]: expected trait, found function `generator_capture`
  --> /home/zetanumbers/dev/rust/tests/ui/parallel-rustc/query-cycle-issue-129912.rs:65:29
   |
LL | fn generator_hold() -> impl generator_capture {
   |                             ^^^^^^^^^^^^^^^^^ not a trait

error[E0658]: yield syntax is experimental
  --> /home/zetanumbers/dev/rust/tests/ui/parallel-rustc/query-cycle-issue-129912.rs:56:9
   |
LL |         yield;
   |         ^^^^^
   |
   = note: see issue #43122 <https://github.com/rust-lang/rust/issues/43122> for more information
   = help: add `#![feature(coroutines)]` to the crate attributes to enable
   = note: this compiler was built on YYYY-MM-DD; consider upgrading it if it is out of date

error[E0658]: yield syntax is experimental
  --> /home/zetanumbers/dev/rust/tests/ui/parallel-rustc/query-cycle-issue-129912.rs:68:9
   |
LL |         yield;
   |         ^^^^^
   |
   = note: see issue #43122 <https://github.com/rust-lang/rust/issues/43122> for more information
   = help: add `#![feature(coroutines)]` to the crate attributes to enable
   = note: this compiler was built on YYYY-MM-DD; consider upgrading it if it is out of date

error[E0562]: `impl Trait` is not allowed in cast expression types
  --> /home/zetanumbers/dev/rust/tests/ui/parallel-rustc/query-cycle-issue-129912.rs:24:22
   |
LL |     &ptr() as *const impl Sized
   |                      ^^^^^^^^^^
   |
   = note: `impl Trait` is only allowed in arguments and return types of functions and methods

error[E0658]: yield syntax is experimental
  --> /home/zetanumbers/dev/rust/tests/ui/parallel-rustc/query-cycle-issue-129912.rs:56:9
   |
LL |         yield;
   |         ^^^^^
   |
   = note: see issue #43122 <https://github.com/rust-lang/rust/issues/43122> for more information
   = help: add `#![feature(coroutines)]` to the crate attributes to enable
   = note: this compiler was built on YYYY-MM-DD; consider upgrading it if it is out of date

error: `yield` can only be used in `#[coroutine]` closures, or `gen` blocks
  --> /home/zetanumbers/dev/rust/tests/ui/parallel-rustc/query-cycle-issue-129912.rs:56:9
   |
LL |         yield;
   |         ^^^^^
   |
help: use `#[coroutine]` to make this closure a coroutine
   |
LL |     #[coroutine] move || {
   |     ++++++++++++

error[E0658]: yield syntax is experimental
  --> /home/zetanumbers/dev/rust/tests/ui/parallel-rustc/query-cycle-issue-129912.rs:68:9
   |
LL |         yield;
   |         ^^^^^
   |
   = note: see issue #43122 <https://github.com/rust-lang/rust/issues/43122> for more information
   = help: add `#![feature(coroutines)]` to the crate attributes to enable
   = note: this compiler was built on YYYY-MM-DD; consider upgrading it if it is out of date

error: `yield` can only be used in `#[coroutine]` closures, or `gen` blocks
  --> /home/zetanumbers/dev/rust/tests/ui/parallel-rustc/query-cycle-issue-129912.rs:68:9
   |
LL |         yield;
   |         ^^^^^
   |
help: use `#[coroutine]` to make this closure a coroutine
   |
LL |     #[coroutine] move || {
   |     ++++++++++++

error[E0121]: the placeholder `_` is not allowed within types on item signatures for return types
  --> /home/zetanumbers/dev/rust/tests/ui/parallel-rustc/query-cycle-issue-129912.rs:19:15
   |
LL | fn array() -> _ {
   |               ^ not allowed in type signatures

error[E0121]: the placeholder `_` is not allowed within types on item signatures for return types
  --> /home/zetanumbers/dev/rust/tests/ui/parallel-rustc/query-cycle-issue-129912.rs:45:21
   |
LL | fn closure_sig() -> _ {
   |                     ^ not allowed in type signatures

error[E0121]: the placeholder `_` is not allowed within types on item signatures for return types
  --> /home/zetanumbers/dev/rust/tests/ui/parallel-rustc/query-cycle-issue-129912.rs:23:13
   |
LL | fn ptr() -> _ {
   |             ^ not allowed in type signatures

thread 'rustc query cycle handler' panicked at compiler/rustc_query_system/src/query/job.rs:535:9:
deadlock detected as we're unable to find a query cycle to break
current query map:
{
    QueryJobId(
        9312,
    ): QueryJobInfo {
        query: QueryStackFrame {
            description: "checking that `array` is well-formed",
            span: Some(
                Span {
                    lo: BytePos(
                        457,
                    ),
                    hi: BytePos(
                        472,
                    ),
                    ctxt: #0,
                },
            ),
            def_id: Some(
                DefId(0:7),
            ),
            def_kind: Some(
                Fn,
            ),
            ty_def_id: None,
            dep_kind: DepKind {
                variant: 200,
            },
            hash: 2960150798538823019,
        },
        job: QueryJob {
            id: QueryJobId(
                9312,
            ),
            span: Span {
                lo: BytePos(
                    0,
                ),
                hi: BytePos(
                    0,
                ),
                ctxt: #0,
            },
            parent: Some(
                QueryJobId(
                    9311,
                ),
            ),
            latch: None,
        },
    },
    QueryJobId(
        9416,
    ): QueryJobInfo {
        query: QueryStackFrame {
            description: "checking that `closure_sig` is well-formed",
            span: Some(
                Span {
                    lo: BytePos(
                        811,
                    ),
                    hi: BytePos(
                        832,
                    ),
                    ctxt: #0,
                },
            ),
            def_id: Some(
                DefId(0:18),
            ),
            def_kind: Some(
                Fn,
            ),
            ty_def_id: None,
            dep_kind: DepKind {
                variant: 200,
            },
            hash: 16075210718503077762,
        },
        job: QueryJob {
            id: QueryJobId(
                9416,
            ),
            span: Span {
                lo: BytePos(
                    0,
                ),
                hi: BytePos(
                    0,
                ),
                ctxt: #0,
            },
            parent: Some(
                QueryJobId(
                    9311,
                ),
            ),
            latch: None,
        },
    },
    QueryJobId(
        9327,
    ): QueryJobInfo {
        query: QueryStackFrame {
            description: "type-checking `ptr`",
            span: Some(
                Span {
                    lo: BytePos(
                        492,
                    ),
                    hi: BytePos(
                        505,
                    ),
                    ctxt: #0,
                },
            ),
            def_id: Some(
                DefId(0:8),
            ),
            def_kind: Some(
                Fn,
            ),
            ty_def_id: None,
            dep_kind: DepKind {
                variant: 112,
            },
            hash: 12849655396065928762,
        },
        job: QueryJob {
            id: QueryJobId(
                9327,
            ),
            span: Span {
                lo: BytePos(
                    0,
                ),
                hi: BytePos(
                    0,
                ),
                ctxt: #0,
            },
            parent: Some(
                QueryJobId(
                    9326,
                ),
            ),
            latch: None,
        },
    },
    QueryJobId(
        9427,
    ): QueryJobInfo {
        query: QueryStackFrame {
            description: "type-checking `closure_sig`",
            span: Some(
                Span {
                    lo: BytePos(
                        811,
                    ),
                    hi: BytePos(
                        832,
                    ),
                    ctxt: #0,
                },
            ),
            def_id: Some(
                DefId(0:18),
            ),
            def_kind: Some(
                Fn,
            ),
            ty_def_id: None,
            dep_kind: DepKind {
                variant: 112,
            },
            hash: 15655468259036780772,
        },
        job: QueryJob {
            id: QueryJobId(
                9427,
            ),
            span: Span {
                lo: BytePos(
                    0,
                ),
                hi: BytePos(
                    0,
                ),
                ctxt: #0,
            },
            parent: Some(
                QueryJobId(
                    9426,
                ),
            ),
            latch: None,
        },
    },
    QueryJobId(
        9121,
    ): QueryJobInfo {
        query: QueryStackFrame {
            description: "running analysis passes on this crate",
            span: Some(
                Span {
                    lo: BytePos(
                        0,
                    ),
                    hi: BytePos(
                        0,
                    ),
                    ctxt: #0,
                },
            ),
            def_id: None,
            def_kind: None,
            ty_def_id: None,
            dep_kind: DepKind {
                variant: 25,
            },
            hash: 5462938028500440564,
        },
        job: QueryJob {
            id: QueryJobId(
                9121,
            ),
            span: Span {
                lo: BytePos(
                    0,
                ),
                hi: BytePos(
                    0,
                ),
                ctxt: #0,
            },
            parent: None,
            latch: None,
        },
    },
    QueryJobId(
        9671,
    ): QueryJobInfo {
        query: QueryStackFrame {
            description: "type-checking `array`",
            span: Some(
                Span {
                    lo: BytePos(
                        457,
                    ),
                    hi: BytePos(
                        472,
                    ),
                    ctxt: #0,
                },
            ),
            def_id: Some(
                DefId(0:7),
            ),
            def_kind: Some(
                Fn,
            ),
            ty_def_id: None,
            dep_kind: DepKind {
                variant: 112,
            },
            hash: 17306232884367025954,
        },
        job: QueryJob {
            id: QueryJobId(
                9671,
            ),
            span: Span {
                lo: BytePos(
                    0,
                ),
                hi: BytePos(
                    0,
                ),
                ctxt: #0,
            },
            parent: Some(
                QueryJobId(
                    9659,
                ),
            ),
            latch: None,
        },
    },
    QueryJobId(
        9311,
    ): QueryJobInfo {
        query: QueryStackFrame {
            description: "checking that types are well-formed in top-level module",
            span: Some(
                Span {
                    lo: BytePos(
                        241,
                    ),
                    hi: BytePos(
                        1405,
                    ),
                    ctxt: #0,
                },
            ),
            def_id: Some(
                DefId(0:0),
            ),
            def_kind: Some(
                Mod,
            ),
            ty_def_id: None,
            dep_kind: DepKind {
                variant: 110,
            },
            hash: 16587930349606688465,
        },
        job: QueryJob {
            id: QueryJobId(
                9311,
            ),
            span: Span {
                lo: BytePos(
                    0,
                ),
                hi: BytePos(
                    0,
                ),
                ctxt: #0,
            },
            parent: Some(
                QueryJobId(
                    9121,
                ),
            ),
            latch: None,
        },
    },
    QueryJobId(
        9659,
    ): QueryJobInfo {
        query: QueryStackFrame {
            description: "computing function signature of `array`",
            span: Some(
                Span {
                    lo: BytePos(
                        457,
                    ),
                    hi: BytePos(
                        472,
                    ),
                    ctxt: #0,
                },
            ),
            def_id: Some(
                DefId(0:7),
            ),
            def_kind: Some(
                Fn,
            ),
            ty_def_id: None,
            dep_kind: DepKind {
                variant: 98,
            },
            hash: 7546817883341146202,
        },
        job: QueryJob {
            id: QueryJobId(
                9659,
            ),
            span: Span {
                lo: BytePos(
                    0,
                ),
                hi: BytePos(
                    0,
                ),
                ctxt: #0,
            },
            parent: Some(
                QueryJobId(
                    9312,
                ),
            ),
            latch: Some(
                QueryLatch {
                    info: Mutex {
                        data: QueryLatchInfo {
                            complete: true,
                            waiters: [],
                        },
                    },
                },
            ),
        },
    },
    QueryJobId(
        9326,
    ): QueryJobInfo {
        query: QueryStackFrame {
            description: "computing function signature of `ptr`",
            span: Some(
                Span {
                    lo: BytePos(
                        492,
                    ),
                    hi: BytePos(
                        505,
                    ),
                    ctxt: #0,
                },
            ),
            def_id: Some(
                DefId(0:8),
            ),
            def_kind: Some(
                Fn,
            ),
            ty_def_id: None,
            dep_kind: DepKind {
                variant: 98,
            },
            hash: 13208712352682470133,
        },
        job: QueryJob {
            id: QueryJobId(
                9326,
            ),
            span: Span {
                lo: BytePos(
                    0,
                ),
                hi: BytePos(
                    0,
                ),
                ctxt: #0,
            },
            parent: Some(
                QueryJobId(
                    9316,
                ),
            ),
            latch: Some(
                QueryLatch {
                    info: Mutex {
                        data: QueryLatchInfo {
                            complete: true,
                            waiters: [],
                        },
                    },
                },
            ),
        },
    },
    QueryJobId(
        9426,
    ): QueryJobInfo {
        query: QueryStackFrame {
            description: "computing function signature of `closure_sig`",
            span: Some(
                Span {
                    lo: BytePos(
                        811,
                    ),
                    hi: BytePos(
                        832,
                    ),
                    ctxt: #0,
                },
            ),
            def_id: Some(
                DefId(0:18),
            ),
            def_kind: Some(
                Fn,
            ),
            ty_def_id: None,
            dep_kind: DepKind {
                variant: 98,
            },
            hash: 7436694756099374155,
        },
        job: QueryJob {
            id: QueryJobId(
                9426,
            ),
            span: Span {
                lo: BytePos(
                    0,
                ),
                hi: BytePos(
                    0,
                ),
                ctxt: #0,
            },
            parent: Some(
                QueryJobId(
                    9416,
                ),
            ),
            latch: Some(
                QueryLatch {
                    info: Mutex {
                        data: QueryLatchInfo {
                            complete: true,
                            waiters: [],
                        },
                    },
                },
            ),
        },
    },
    QueryJobId(
        9316,
    ): QueryJobInfo {
        query: QueryStackFrame {
            description: "checking that `ptr` is well-formed",
            span: Some(
                Span {
                    lo: BytePos(
                        492,
                    ),
                    hi: BytePos(
                        505,
                    ),
                    ctxt: #0,
                },
            ),
            def_id: Some(
                DefId(0:8),
            ),
            def_kind: Some(
                Fn,
            ),
            ty_def_id: None,
            dep_kind: DepKind {
                variant: 200,
            },
            hash: 15152811524914447862,
        },
        job: QueryJob {
            id: QueryJobId(
                9316,
            ),
            span: Span {
                lo: BytePos(
                    0,
                ),
                hi: BytePos(
                    0,
                ),
                ctxt: #0,
            },
            parent: Some(
                QueryJobId(
                    9311,
                ),
            ),
            latch: None,
        },
    },
}
stack backtrace:
   0: rust_begin_unwind
             at ./library/std/src/panicking.rs:665:5
   1: core::panicking::panic_fmt
             at ./library/core/src/panicking.rs:74:14
   2: rustc_query_system::query::job::break_query_cycles
             at ./compiler/rustc_query_system/src/query/job.rs:535:9
   3: rustc_interface::util::run_in_thread_pool_with_globals::<rustc_interface::interface::run_compiler<core::result::Result<(), rustc_span::ErrorGuaranteed>, rustc_driver_impl::run_compiler::{closure#0}>::{closure#1}, core::result::Result<(), rustc_span::ErrorGuaranteed>>::{closure#2}::{closure#1}
             at ./compiler/rustc_interface/src/util.rs:191:21
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

error: the compiler unexpectedly panicked. this is a bug.

note: we would appreciate a bug report: https://github.com/rust-lang/rust/issues/new?labels=C-bug%2C+I-ICE%2C+T-compiler&template=ice.md

note: please make sure that you have updated to the latest nightly

note: rustc 1.84.0-dev running on x86_64-unknown-linux-gnu

note: compiler flags: -Z simulate-remapped-rust-src-base=/rustc/FAKE_PREFIX -Z translate-remapped-path-to-local-path=no -Z ignore-directory-in-diagnostics-source-blocks=/home/zetanumbers/.cargo -Z ignore-directory-in-diagnostics-source-blocks=/home/zetanumbers/dev/rust/vendor -C codegen-units=1 -Z ui-testing -Z deduplicate-diagnostics=no -Z write-long-types-to-disk=no -C strip=debuginfo -C prefer-dynamic -C rpath -C debuginfo=0 -Z threads=16

query stack during panic:
end of query stack
query cycle handler thread panicked, aborting process
------------------------------------------

[jieyouxu]

@SparrowLii this PR is still blocked on #137731 merging first, then this PR will need to rebase right?

[jieyouxu]

@rustbot blocked (#137731)

[lqd]

Do we need a new directive, or does this make sense as a dedicated test suite instead?

//@ parallel-front-end-robustness changes how the test is executed and verified, ignoring its output, so it's kinda "dangerous" to add it to an existing UI test: the test output and blessed expectations could diverge without the test failing. It makes perfect sense to have dedicated tests with it, like the ones in this PR, maybe more so than a directive added to existing tests.

So if this is only for new, dedicated tests, the opt-in directive mechanism may be less interesting than having a different dedicated test suite: the parallel test suite where all tests are run under this new execution and verification setup, much like the incremental suite sets up incremental compilation and verification for the tests it contains.

This would allow for easy expansion in the future, for example:

• by not only having multiple iterations, but also dynamically change the number of threads for each execution of a given test, instead of hard-coding -Zthreads in the test compile-flags
• trigger adversarial conditions for robustness (something like query fuzzing with reorderings and delays, think chaos monkey or loom)
• or run tests under other tools or sanitizers

Maybe it could also be interesting to be able to configure the iteration count without rebuilding compiletest, maybe an env var to control this for the entire suite. Maybe a directive could also be useful per test, I don't know. These would allow one to crank up the number of iterations if they need to try to reproduce a particular issue: locally I've had to do this increase over the 50 iterations count in the PR, to reproduce some of the known deadlocks more easily. I guess it could allow CI some control to spend more or less time in these tests as well if we want to expand coverage (though the question of how to reliably test and benchmark the parallel compiler is a different thorny issue altogether).

What do you think @SparrowLii and @jieyouxu?

[jieyouxu]

Do we need a new directive, or does this make sense as a dedicated test suite instead?

//@ parallel-front-end-robustness changes how the test is executed and verified, ignoring its output, so it's kinda "dangerous" to add it to an existing UI test: the test output and blessed expectations could diverge without the test failing. It makes perfect sense to have dedicated tests with it, like the ones in this PR, maybe more so than a directive added to existing tests.

I would be fine with some kind of parallel-frontend test suite that's Mode::Ui but a separate test suite, sure. I haven't looked at the changes in this PR too thoroughly yet since it was blocked on prerequisite PRs merging.

So if this is only for new, dedicated tests, the opt-in directive mechanism may be less interesting than having a different dedicated test suite: the parallel test suite where all tests are run under this new execution and verification setup, much like the incremental suite sets up incremental compilation and verification for the tests it contains.

This would allow for easy expansion in the future, for example:

* by not only having multiple iterations, but also dynamically change the number of threads for each execution of a given test, instead of hard-coding `-Zthreads` in the test compile-flags

* trigger adversarial conditions for robustness (something like query fuzzing with reorderings and delays, think [chaos monkey](https://netflix.github.io/chaosmonkey/) or [loom](https://github.com/tokio-rs/loom?rgh-link-date=2025-03-04T09%3A14%3A13Z))

* or run tests under other tools or sanitizers

Yes. I do agree that a separate test suite could be better to serve the purposes of parallel front-end specifically and not have to special-case a ui test subdir or shoehorn existing ui test suite to cater to parallel front-end tests.

In a sense, this is also forwards-compatible. Once we are sufficiently confident in the parallel front-end to be very likely to not cause new test failures, we can adjust the existing ui test suite to also run with a configuration similar to the new parallel-frontend test suite, and potentially merge the two.

Maybe it could also be interesting to be able to configure the iteration count without rebuilding compiletest, maybe an env var to control this for the entire suite. Maybe a directive could also be useful per test, I don't know. These would allow one to crank up the number of iterations if they need to try to reproduce a particular issue: locally I've had to do this increase over the 50 iterations count in the PR, to reproduce some of the known deadlocks more easily. I guess it could allow CI some control to spend more or less time in these tests as well if we want to expand coverage (though the question of how to reliably test and benchmark the parallel compiler is a different thorny issue altogether).

Yes, that sounds like a use case one might want to handle. For instance, you may even want to configure this via config.toml or sth.

[SparrowLii]

I agree. Parallel front end testing is very different from the current UI test cases. We should build a dedicated test suite so that it will not be abused in other UI tests in the future or make the UI test suite more and more bloated.

I think we can create a new parallel-rustc folder under tests.

[bors]

☔ The latest upstream changes (presumably #138058) made this pull request unmergeable. Please resolve the merge conflicts.