Contradiction between public results and bert_example.ipynb

[Bachstelze]

The left bar in figure 4 a) in the paper shows the relative predicted probability for the two possible pronouns, “his” and “her” with a correct prediction to "his" for the input "the author talked to sara about [MASK] book". bert_example.ipynb is giving the opposite result for that sentence:

[Bachstelze]

This false result is also produced by the pipelined huggingface inference API:

[Bachstelze]

With bert-base-case the results are also wrong:

Also without punctuation:

[HelloWorldLTY]

I think it make senes because during the plot the authors only keep his and her to display. In the plot you showed, the probability of her is always higher than his.

[Bachstelze]

@HelloWorldLTY What do you mean by "during the plot the authors only keep his and her to display"?
The contradicting result is only about this input sentence "the author talked to sara about [MASK] book".

[Bachstelze]

The reason for the different results is probably different model sizes. BERT-large gives the same result as in the paper: