Reproducibility

[emilycantrell]

Tommy wrote:

On Sept. 16, I messaged you in this slack chat that the cv performance of the winning model setup was 0.8007, but it seems that you got 0.794 according to your ablation tests?

In January we'll figure out what went wrong.

Note that when we do the runs for casual testing purposes, we haven't been running the full workflow -- rather, we've been running just the relevant sections of the code, so maybe this is the problem. (But we do have a seed set at the relevant location which we thought would resolve this.)