HuggingFace LogitsProcessor to accept also list of compiled grammars
#276
Comments
Ubospica
pushed a commit
that referenced
this issue
Apr 4, 2025
#275) My use case is that I need to follow a different grammar for each row in a batch. I couldn't find a way to do this natively with current API, so started hacking and came up with the changes in the PR. I'm not sure about the broader context, so would appreciate a comment if this approach indeed makes sense. ``` class CountryPoland(ClassSchema): country: Literal["Poland"] class CountryGermany(ClassSchema): country: Literal["Germany"] tokenizer_info = xgr.TokenizerInfo.from_huggingface(tokenizer, vocab_size=config.vocab_size) grammar_compiler = xgr.GrammarCompiler(tokenizer_info) compiled_grammars = [ grammar_compiler.compile_json_schema(CountryPoland), grammar_compiler.compile_json_schema(CountryGermany), ] logit_processor = LogitsProcessor(compiled_grammars) logit_processor = LogitsProcessorList([logit_processor]) model.generate(**model_inputs, max_new_tokens=512, logits_processor=logit_processor) ``` [Linked issue](#276).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Thanks for this amazing library!
I'd like to be able to provide different compiled grammars within single batch. I couldn't find a way to do this and I believe changes proposed in this PR are one way to enable that.
I'd appreciate your comment and if I should pursue work on this PR. Thanks!
The text was updated successfully, but these errors were encountered: