Node Level Scaling on KNL (on Cori)

On this page, we track the state of node-level scaling on KNL (on Cori).

Current state

• Drop in parallel efficiency even for only 2 threads
• Runtime only decreases for ~30 threads, then it starts to increase again
• Level 5 results will be added soon which will also include idle rate measurements
• Getting an out-of-memory for level 5 with MCDRAM and level 6 with DRAM

The following graphs display the runtime for different numbers of hpx threads ("-hpx:threads").

For Dominic's model problems:

For the moving star problem:

This corresponds to the parallel efficiency displayed in the next graphs.

For Dominic's problem:

For the moving star problem:

• Experiment with level 6 and one thread ran out of time, messes up parallel efficiency

Console output to individual experiments can be found here.

Reproduce

The following command was used for the moving stars runs:

srun numactl -m 1 ./knl-build/octotiger-Release/octotiger \
-Disableoutput -Problem=moving_star -Max_level=${level} \
 -Stopstep=0 --hpx:threads=${threads} \
--hpx:ini=hpx.stacks.small_size=0xC00000 -Ihpx.stacks.use_guard_pages=0 \
--hpx:print-bind --hpx:print-counter /threads{locality#*/total}/idle-rate \ 
> results/${name}_N${SLURM_NNODES}_t${threads}_l${level}_m1 2>&1

And for Dominic's problems:

srun numactl -m 1 ./knl-build/octotiger-Release/octotiger -Disableoutput -Problem=dwd \
-Max_level=${level} -Xscale=4.0 -Eos=wd -Angcon=1 -Stopstep=0.01 \
--hpx:threads=${threads} -Restart=restart${level}.chk \
--hpx:ini=hpx.stacks.small_size=0xC0000 -Ihpx.stacks.use_guard_pages=0 \
--hpx:print-bind --hpx:print-counter /threads{locality#*/total}/idle-rate \
  > results/${name}_N${SLURM_NNODES}_t${threads}_l${level}_m1 2>&1