Add Cloud RMs

[natolambert]

See @zankner's repo https://github.com/zankner/CLoud, RM's that think out loud!

[scottsuk0306]

Hi @natolambert, I recently trained a generative RM (prometheus-eval/prometheus-RM-Llama-8B-v1.0) based on the CLoud code base and l think the inference using the huggingface transformers can be easily integrated to existing code of reward-bench. Can I try working on this?

[natolambert]

@scottsuk0306 yes please! I've been curious for a bit and we're building new datasets now

FYI -- my issue was the tokenization timing was specific and would require a bunch of handling or a refactor. Lmk if you figure out something clever.

[zankner]

Hi sorry realized I never followed up on this. @scottsuk0306 @natolambert we support HF inference in the CLoud repo which I can also add (hf-inference). However, HF generate is extremely slow. More recently we added vllm-support but I'm not sure whether that would be easy to support in your repo, if you have any thoughts.

[natolambert]

We could try running Cloud RM with the "generative" pipeline, which is a bit different.
Otherwise, I feel like it isn't needed to use VLLM, unless we can figure out how to wrap it in the same abstraction -- which actually may be possible, because we just do some general model loading.

Either way, curious where we end up.