Skip to content

Commit f6cc33e

Browse files
borg323PikaCat-OuO
authored andcommitted
upstream the rescorer (LeelaChessZero#1957)
* Add a mode to turn lc0 into a chunk data rescorer powered by Tablebase. * Add some stats. * Add secondary rescoring using wdl to reduce back propigation of blunders a bit. * Add policy distribution adjust support to rescorer. * Track the game outcomes, and the change to the start of the game * Add DTZ based assistance for secondary rescoring. * Change move count to a moves remaining to potentially use for modulating target value. * Use DTZ for pawnless 3 piece positions as a substitue for DTM to adjust move_count to be more correct * another fix. * More fixing. * Getting things compiling again. * Make rescorer more obvious. * reorder to match struct order. * Actually update the version when converting to v4 format. * Implement the threading support. * Fix compilation issues on some compilers. * More compilation fixing. * Fix off by one. * Add support for root probe policy boosting for minimum dtz in winning positions. * Fix test compile. * Fix missing option. * Add a counter. * Log if policy boost is for a move labelled illegal. * Add a histogram for total amount of boosted policy per boosted position. * Distribute boost rather than apply to all - also log before and after dists. * Add gaviotatb code for later use in dtm_boost * Fix compile issue on linux. * Prepare logic for dtm policy boost. * Load gaviota tb if specified. * Probe gaviota to decide which 'safe' moves are most deserving of boost based on dtm. * First attempt at supporting arbitrary starting point training data for rescorer. * Fix missing brackets. * Some fixes. * Avoid crashes from walking history before start of provided game information. * Some more merge fixes. * Fix some formatting. * Only process .gz files, don't crash out on invalid files, don't create output until input has been read. * Don't keep partially valid files. * Add basic range validation for input data. * Don't create writer any earlier than needed. * Fix decoding castling moves for the new Move format. * Validate game moves for legality. * Also log illegal move if it passes probability check but fails the real check. * Fix another merge error. * Compile fix for linux. * Plies left in rescorer (LeelaChessZero#1) * Rescore move_count using Gaviota TBs * Fix lczero-common commit * Add condition for Gaviota move_count rescoring * Post merge fixup for the kings/knights change in board. * Rescore tb v5 (LeelaChessZero#2) * Make lc0 output v5 training data. * Finish merge of v5 data into rescorer tb. * Fixes for rescoring v4 data. * Revert some unneeded formatting changes. * Support FRC input_format in rescoring. * Add some very important missing break statements... * Fix merge. * Change movement decode to not rely on there being any history planes filled in. Since that will not always be the case for input type 3. * Minimum changes to make it compile again post merge. * Input format 3 support. * Fix data range checks were incorrect for format 3 and 2. * Fix up bugs with chess 960 castle moves that leave a rook or king in place. * Post merge compile fixups for renames. * Add support for hectoplies and hectoplies armageddon to validate, and fixup the merge of latest code. * More fixes for type 4 and 132. * Add input format conversion support to rescorer. * Better match for training. * Add canonical v2 format to rescorer. * Add a utility for substituting policy from higher quality data into main data. * Fix missing option and add some commented out diagnostic code. * More cleanup in comments. * Handle empty policy-substitutions dir and input dir better. * Don't keep chunks that are marked as not for training. * More fixes for handling files with placeholder chunks. * Add 'deblunderer' Completely untested... * Fix some bugs in deblunder. * simplify windows rescorer build (LeelaChessZero#4) Co-authored-by: borg323 <borg323@users.noreply.github.com> * Tweak windows build file. * Some updates for writer.h/cc for v6 * Update rescorer loop.cc for V6. * Some additional validations to do with played_idx/best_idx. * make appveyor build the rescorer (LeelaChessZero#7) Co-authored-by: borg323 <borg323@users.noreply.github.com> * subproject for gaviota tb files (LeelaChessZero#8) Co-authored-by: borg323 <borg323@users.noreply.github.com> * 'Fix' for build on windows Probably should be fixed some other way... * Fix my breakage. (LeelaChessZero#9) * Update loop.cc * Update meson.build * Use the v6 field played_q to do a more direct blunder rescoring (LeelaChessZero#5) * included the issue 1308 deblunder mechanism in loop.cc * blunder detection now acts on missed proven wins and unforced proven losses * added comment on missing activeM * removed probabilistic randomization of result rescorer and worked with v6 data instead * included moves left rescore, removed unneeded options * doubled code not needed as final positions aren't special * changed appveyor script to hopefully build rescorer.sln * reverted failed attempt at fixing appveyor * included minimal std::cout for blunders * included blunder counter, added comment to visits v6 data checking * checking for bit 3 of invariance info to make sure best_q is a proven win * Fix v5 upgrading for decisive games. * Additional safety. * Add missing brackets. * don't keep the first TB position for the deblundering pass. (LeelaChessZero#10) * included the issue 1308 deblunder mechanism in loop.cc * blunder detection now acts on missed proven wins and unforced proven losses * added comment on missing activeM * removed probabilistic randomization of result rescorer and worked with v6 data instead * included moves left rescore, removed unneeded options * doubled code not needed as final positions aren't special * changed appveyor script to hopefully build rescorer.sln * reverted failed attempt at fixing appveyor * included minimal std::cout for blunders * included blunder counter, added comment to visits v6 data checking * checking for bit 3 of invariance info to make sure best_q is a proven win * don't keep the first TB position for rescorer * change recorer logo (LeelaChessZero#11) Co-authored-by: borg323 <borg323@users.noreply.github.com> * Make the deblunder transition soft through a width parameter (LeelaChessZero#13) * included the issue 1308 deblunder mechanism in loop.cc * blunder detection now acts on missed proven wins and unforced proven losses * added comment on missing activeM * removed probabilistic randomization of result rescorer and worked with v6 data instead * included moves left rescore, removed unneeded options * doubled code not needed as final positions aren't special * changed appveyor script to hopefully build rescorer.sln * reverted failed attempt at fixing appveyor * included minimal std::cout for blunders * included blunder counter, added comment to visits v6 data checking * checking for bit 3 of invariance info to make sure best_q is a proven win * don't keep the first TB position for rescorer * added a deblunder width parameter to allow a soft transition * clang formatting * resolve merge conflict * Add nnue plain file output (LeelaChessZero#12) * GetFen() from pr834 * first version of nnue output * flag to delete fils * address review comments * support pre v6 data * fix sign * correct nnue data misunderstanding Co-authored-by: borg323 <borg323@users.noreply.github.com> * fix copy-paste error (LeelaChessZero#15) Co-authored-by: borg323 <borg323@users.noreply.github.com> * add -t flag (LeelaChessZero#16) Co-authored-by: borg323 <borg323@users.noreply.github.com> * Post merge fixes. * Missed cleanup. * Fix input format change bug that can corrupt played_idx and best_idx * Post merge fixes. * fix merge * remove unnecessary options * split out rescore loop * minimize rescorer build * merge rescorer with master * minimize syzygy diff --------- Co-authored-by: Tilps <tilps@users.noreply.github.com> Co-authored-by: Henrik Forstén <henrik.forsten@gmail.com> Co-authored-by: borg323 <borg323@users.noreply.github.com> Co-authored-by: Naphthalin <40385638+Naphthalin@users.noreply.github.com> (cherry picked from commit 738c4aa)
1 parent b9f5c67 commit f6cc33e

10 files changed

+1020
-855
lines changed

build_rescorer.cmd

+35
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
@echo off
2+
setlocal
3+
4+
echo Deleting build directory:
5+
rd /s build
6+
7+
set CC=cl
8+
set CXX=cl
9+
set CC_LD=link
10+
set CXX_LD=link
11+
12+
if exist "C:\Program Files\Microsoft Visual Studio\2022" (
13+
where /q cl
14+
if errorlevel 1 call "C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64
15+
set backend=vs2022
16+
) else if exist "C:\Program Files (x86)\Microsoft Visual Studio\2019" (
17+
where /q cl
18+
if errorlevel 1 call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64
19+
set backend=vs2019
20+
) else (
21+
where /q cl
22+
if errorlevel 1 call "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Auxiliary\Build\vcvarsall.bat" amd64
23+
set backend=vs2017
24+
)
25+
26+
meson build --backend %backend% --buildtype release -Drescorer=true -Dlc0=false -Dgtest=false -Ddefault_library=static
27+
28+
if errorlevel 1 exit /b
29+
30+
pause
31+
32+
cd build
33+
34+
msbuild /m /p:Configuration=Release /p:Platform=x64 /p:WholeProgramOptimization=true ^
35+
/p:PreferredToolArchitecture=x64 rescorer.sln /filelogger

meson.build

+44-26
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@
1616

1717
project('lc0', 'cpp',
1818
default_options : ['cpp_std=c++17', 'b_ndebug=if-release', 'warning_level=3', 'b_lto=true', 'b_vscrt=mt'],
19-
meson_version: '>=0.54')
19+
meson_version: '>=0.55')
2020

2121
cc = meson.get_compiler('cpp')
2222

@@ -57,6 +57,7 @@ endif
5757

5858
# Files to compile.
5959
deps = []
60+
common_files = []
6061
files = []
6162
includes = []
6263
has_backends = false
@@ -95,7 +96,7 @@ pb_files = [
9596
gen.process('libs/lczero-common/proto/net.proto',
9697
preserve_path_from : meson.current_source_dir() + '/libs/lczero-common/')
9798
]
98-
files += pb_files
99+
common_files += pb_files
99100

100101
# Extract git short revision.
101102
short_rev = 'unknown'
@@ -152,18 +153,36 @@ files += gen_proto_src.process('src/neural/onnx/onnx.proto',
152153
#############################################################################
153154
## Main files
154155
#############################################################################
155-
files += [
156-
'src/benchmark/backendbench.cc',
157-
'src/benchmark/benchmark.cc',
156+
common_files += [
158157
'src/chess/bitboard.cc',
159158
'src/chess/board.cc',
160159
'src/chess/position.cc',
161160
'src/chess/uciloop.cc',
161+
'src/mcts/node.cc',
162+
'src/neural/decoder.cc',
163+
'src/neural/encoder.cc',
164+
'src/trainingdata/reader.cc',
165+
'src/trainingdata/trainingdata.cc',
166+
'src/trainingdata/writer.cc',
167+
'src/utils/commandline.cc',
168+
'src/utils/configfile.cc',
169+
'src/utils/esc_codes.cc',
170+
'src/utils/files.cc',
171+
'src/utils/logging.cc',
172+
'src/utils/optionsdict.cc',
173+
'src/utils/optionsparser.cc',
174+
'src/utils/random.cc',
175+
'src/utils/string.cc',
176+
'src/version.cc',
177+
]
178+
179+
files += [
180+
'src/benchmark/backendbench.cc',
181+
'src/benchmark/benchmark.cc',
162182
'src/engine.cc',
163183
'src/lc0ctl/describenet.cc',
164184
'src/lc0ctl/leela2onnx.cc',
165-
'src/lc0ctl/onnx2leela.cc',
166-
'src/mcts/node.cc',
185+
'src/lc0ctl/onnx2leela.cc',
167186
'src/mcts/params.cc',
168187
'src/mcts/search.cc',
169188
'src/mcts/stoppers/alphazero.cc',
@@ -175,8 +194,6 @@ files += [
175194
'src/mcts/stoppers/stoppers.cc',
176195
'src/mcts/stoppers/timemgr.cc',
177196
'src/neural/cache.cc',
178-
'src/neural/decoder.cc',
179-
'src/neural/encoder.cc',
180197
'src/neural/factory.cc',
181198
'src/neural/loader.cc',
182199
'src/neural/network_check.cc',
@@ -193,22 +210,9 @@ files += [
193210
'src/selfplay/game.cc',
194211
'src/selfplay/loop.cc',
195212
'src/selfplay/tournament.cc',
196-
'src/trainingdata/reader.cc',
197-
'src/trainingdata/trainingdata.cc',
198-
'src/trainingdata/writer.cc',
199-
'src/utils/commandline.cc',
200-
'src/utils/configfile.cc',
201-
'src/utils/esc_codes.cc',
202-
'src/utils/files.cc',
203213
'src/utils/histogram.cc',
204-
'src/utils/logging.cc',
205214
'src/utils/numa.cc',
206-
'src/utils/optionsdict.cc',
207-
'src/utils/optionsparser.cc',
208-
'src/utils/random.cc',
209-
'src/utils/string.cc',
210215
'src/utils/weights_adapter.cc',
211-
'src/version.cc',
212216
]
213217
includes += include_directories('src')
214218

@@ -218,9 +222,9 @@ deps += dependency('threads')
218222
## Platform specific files
219223
############################################################################
220224
if host_machine.system() == 'windows'
221-
files += 'src/utils/filesystem.win32.cc'
225+
common_files += 'src/utils/filesystem.win32.cc'
222226
else
223-
files += 'src/utils/filesystem.posix.cc'
227+
common_files += 'src/utils/filesystem.posix.cc'
224228
endif
225229

226230
#############################################################################
@@ -642,7 +646,7 @@ if get_option('build_backends')
642646

643647
endif # if get_option('build_backends')
644648

645-
if not has_backends and get_option('build_backends')
649+
if not has_backends and get_option('lc0') and get_option('build_backends')
646650
error('''
647651
648652
No usable computation backends (cudnn/opencl/blas/etc) enabled.
@@ -697,8 +701,22 @@ if get_option('embed')
697701
add_project_arguments('-DEMBED', language : 'cpp')
698702
endif
699703

700-
executable('lc0', 'src/main.cc',
704+
if get_option('lc0')
705+
files += common_files
706+
executable('lc0', 'src/main.cc',
701707
files, include_directories: includes, dependencies: deps, install: true)
708+
endif
709+
710+
#############################################################################
711+
## Rescorer Executable
712+
#############################################################################
713+
714+
if get_option('rescorer')
715+
deps += subproject('gaviotatb').get_variable('gaviotatb_dep')
716+
executable('rescorer', 'src/rescorer_main.cc',
717+
[common_files, 'src/rescorer/rescoreloop.cc'],
718+
include_directories: includes, dependencies: deps, install: true)
719+
endif
702720

703721
#############################################################################
704722
## Tests

meson_options.txt

+11-1
Original file line numberDiff line numberDiff line change
@@ -191,4 +191,14 @@ option('onnx_include',
191191
option('xla',
192192
type: 'boolean',
193193
value: false,
194-
description: 'Enable XLA backend')
194+
description: 'Enable XLA backend')
195+
196+
option('lc0',
197+
type: 'boolean',
198+
value: true,
199+
description: 'Build Lc0')
200+
201+
option('rescorer',
202+
type: 'boolean',
203+
value: false,
204+
description: 'Build rescorer')

src/main.cc

+2-5
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ int main(int argc, const char** argv) {
4545
CERR << EscCodes::Bold() << EscCodes::Red()
4646
<< " _ _";
4747
CERR << R"(|_|\/| |)";
48-
CERR << R"(| /\|_|)"<< EscCodes::Reset() << " v" << GetVersionStr()
48+
CERR << R"(| /\|_|)" << EscCodes::Reset() << " v" << GetVersionStr()
4949
<< " built " << __DATE__;
5050

5151
try {
@@ -65,10 +65,7 @@ int main(int argc, const char** argv) {
6565
CommandLine::RegisterMode("describenet",
6666
"Shows details about the Leela network.");
6767

68-
if (CommandLine::ConsumeCommand("leela2plain")) {
69-
ConvertLoop loop;
70-
loop.RunLoop();
71-
} else if (CommandLine::ConsumeCommand("selfplay")) {
68+
if (CommandLine::ConsumeCommand("selfplay")) {
7269
// Selfplay mode.
7370
SelfPlayLoop loop;
7471
loop.RunLoop();

0 commit comments

Comments
 (0)