Add relevance metrics including pruned tokens to MS Marco ranking track #525
Conversation
kderusso
force-pushed
the
kderusso/msmarco-passage-ranking-iter-from-jim
branch
from
d33ff08 to
1574135
Compare
…e the recall against the original weighted terms query
…NDCG based on a smaller version of the queries used to test performance.
kderusso
force-pushed
the
kderusso/msmarco-passage-ranking-iter-from-jim
branch
from
105b19a to
ac79e54
Compare
| }, | ||
| { | ||
| "name": "pruned-text-expansion-search-maxwand-disabled", |
| }, | ||
| { | ||
| "name": "pruned-weighted-terms-recall-10-10", |
There was a problem hiding this comment.
This file was auto-formatted for correct json indentation. I have left comments starting at the two places where I made changes.
kderusso
changed the title
kderusso
requested review from
afoucret,
demjened,
jimczi,
ioanatia,
Mikep86 and
saikatsarkar056
| def generate_pruned_query(field, query_expansion, boost=1.0): | ||
| return { | ||
| "query": { | ||
| "weighted_tokens": { |
There was a problem hiding this comment.
I've tried running the modified track and noticed weighted_tokens query is not available in 8.11.x. Is this planned for 8.13.0? We need to apply the right versioning as per https://esrally.readthedocs.io/en/stable/track.html#custom-track-repositories. The most recent non-master branch in the track repo is 8.11. We need to bump up to 8.13 before merging this change.
There was a problem hiding this comment.
Correct, this is for 8.13.0.
| pytrec_eval | ||
| numpy |
There was a problem hiding this comment.
Should the presence of this file be mentioned in README? Also, please align version pinning with dependencies section.
Adds recall metrics for query pruning using the weighted tokens query and precomputed ELSER tokens.