"V3TSP.cpp:353: No unmarked edges found in tour" during verilation

[Westwood68]

Verilator 5.032 2025-01-01 rev v5.032
Ubuntu 22.04.5 LTS
AMD Ryzen 5/32GB, AMD Ryzen9/64GB

When verilating a quite complex design (multicore RISC-V SoC), I get the following error:

/opt/eda/verilator/stable/bin/verilator --trace --trace-fst --trace-structs --trace-max-array 128 --trace-threads 2 --threads 16 --clk clk_i --no-timing --Wno-UNOPTTHREADS --Wno-REALCVT --Wno-ASCRANGE --Wno-WIDTHTRUNC --Wno-WIDTHEXPAND --Wno-WIDTHCONCAT --Wno-UNOPTFLAT --Wno-TIMESCALEMOD --Wno-ENUMVALUE --Wno-LATCH --Wno-CASEINCOMPLETE --Wno-MULTIDRIVEN --Wno-CMPCONST --Wno-UNSIGNED --Wno-STMTDLY --Wno-ALWCOMBORDER --Wno-BLKANDNBLK --j 0 -O1 -cc -f ../../../../hw/toplevel/file_list/top_verilator_flist.f --top-module bc64_harness --exe tb_bc64_harness.cpp
%Error: Internal Error: ../V3TSP.cpp:353: No unmarked edges found in tour
... See the manual at https://verilator.org/verilator_doc.html for more assistance.

On machine 1 the error occurs every time, no verilation possible. On machine 2 (same OS), the error occurs also every time, but when starting verilation again, it works. Sometimes but rarely it requires 3 attempts.

The error occurred suddenly, maybe after an update of someting in Ubuntu. I have compiled the same design many times before on the same machines.

I don't have any idea what causes this problem, and I can't boil it down to a certain piece of the RTL code and can't find anything in the documenation.

Any ideas where to start?

[wsnyder]

%Error: Internal Error: ../V3TSP.cpp:353: No unmarked edges found in tour

I haven't seen that before.

That the issue seems unpredictable and changes between systems suggests two primary possibilities:

1. Something is being read from uninitialized memory. Try --debug, and try --valgrind, and try --debug --valgrind.

2. There's a multithreaded race. With -j 0 you are using both --build-jobs 0 and --verilate-jobs 0. Try using just --build-jobs 0. Also note what else is running on the system may affect results; that rerunning works sometimes suggests this is the problem.

You can try to use sv-bugpoint to reduce code to make an example, however as your problem is unpredictable you'll need to run multiple runs for every experiment to avoid it getting confused.

-Wilson

[Westwood68]

Moving from -j 0 to --build-jobs 0 solved the problem. Seems actually something related to multithreaded verilation. Actually I played with the -j option a while ago when trying to speed up the verilation process, and possibly the problem occured then the first time.

I made some further experiments (on the 16core/32threads CPU):
--build-jobs 0 -> works always
--build-jobs 0 --verilate-jobs 0 -> fails always (equivalent to -j 0)
--build-jobs 0 --verilate-jobs 16 -> fails sometimes, unpredictable

Anyway, the problem ist solved. Thank you very much for your support!

[wsnyder]

Good, but there's still a bug hiding in the code with --verilate-jobs; might you be able to use sv-bugpoint to reduce to a test case?

[Westwood68]

Sure, I will try but need some time. Possibly the problem depends on the size of the RTL design? Because this is the biggest design I used in Verilator so far. I will test -j 0 with smaller designs as well and look into sv-bugpoint.