Anomalous streaks in VMC density

[jtkrogel]

Describe the bug

The VMC density is anomalously high in particular voxels. This was discovered by comparing a highly converged VMC density against the density formed by summing contributions from B-spline orbitals (no Jastrow).

The current case is 2x1x1 diamond run with the batched code.

The anomalous cells are highlighted below (taller spikes):

The anomalies occur as streaks in adjacent cells along z. See below:

Files to reproduce are attached:

bug_density_streak.zip

Indices of example streaks are shown below (density is on a 128x64x64 grid):

[ 55   7  32]
 [ 55   7  33]
 [ 55   7  34]
 [ 55   7  36]
 [ 55   7  37]
 [ 55   7  38]
 [ 55   7  39]
 [ 55   7  40]
 [ 55   7  41]
 [ 55   7  42]

 [ 55  38  16]
 [ 55  38  17]
 [ 55  38  18]
 [ 55  38  20]
 [ 55  38  21]
 [ 55  38  23]
 [ 55  38  24]
 [ 55  38  25]

 [ 59   8  12]
 [ 59   8  13]
 [ 59   8  14]
 [ 59   8  15]
 [ 59   8  16]
 [ 59   8  17]
 [ 59   8  18]
 [ 59   8  19]
 [ 59   8  20]
 [ 59   8  22]
 [ 59   8  23]

[ 72  54  44]
 [ 72  54  45]
 [ 72  54  46]
 [ 72  54  47]
 [ 72  54  48]
 [ 72  54  49]
 [ 72  54  50]
 [ 72  54  51]
 [ 72  54  52]
 [ 72  54  53]
 [ 72  54  54]
 [ 72  54  55]
 [ 72  54  56]
 [ 72  54  57]
 [ 72  54  58]
 [ 72  54  59]
 [ 72  54  60]
 [ 72  54  61]
 [ 72  54  62]
 [ 72  54  63]

 [ 83   1  40]
 [ 83   1  41]
 [ 83   1  42]
 [ 83   1  43]
 [ 83   1  44]
 [ 83   1  45]
 [ 83   1  46]
 [ 83   1  47]
 [ 83   8  44]
 [ 83   8  45]
 [ 83   8  46]
 [ 83   8  47]
 [ 83   8  48]
 [ 83   8  49]

To Reproduce

Run the attached files. Make a plot similar to the above.

QMCPACK version:

Git branch: develop
Last git commit: 851f1be
Last git commit date: Fri Oct 18 10:18:32 2024 -0700
Last git commit subject: Merge pull request #5204 from ye-luo/adjust-test

(real cpu build)

Expected behavior

Nice streak-free densities.

System:

Inti at ORNL. 128 core AMD nodes

[prckent]

(Updated)

This is really interesting -- any theories as to the cause? If you have not already I suggest to try varying the Monte Carlo and the grids to try to isolate the cause. e.g. (1) How robust are the streaks if you e.g. change substeps, or add a longer warmup?
Are there any sensitivities to parallelization, e.g. a pure MPI no (single) threads run or run on 1 core? With/without drift? Do the streaks occur in the first or last half of the run or throughout? (2) Does varying the density grids have any effect? e.g. does doubling the grid and rerunning with a fixed seed result in the same spatial pattern?

[jtkrogel]

For reference, the normalization on the density is $\sum_{i\in voxels}\rho_i = N$ (N is electron count).

[jtkrogel]

I will rerun with coarser grids and separately without drift. The runs above were 1MPI per core, single threaded.

[jtkrogel]

NB: analysis of a prior 1x1x1 cell run shows the same type of streaks.

[prckent]

Interesting that any axis is preferred whatsoever.

[jtkrogel]

Further analysis of the 1x1x1 data shows the streaks are present in all block (time) quarters, and that the same cells are involved in each.

[jtkrogel]

NB: Kayahan's DMC data for the 2D database show the same features.

[prckent]

Interesting. Do they have the same orientation? i.e. z axis aligned? And are the supercell z-axis lattice vectors parallel to (0,0,1) in these cases? (It would be very revealing if the streaks were along Cartesian z-axes but the cell vectors were not...).