RuntimeError: Must call ComputeLogits before GenerateNextToken

[ivo-andreev]

Describe the bug
Proposed fix for an error running on CPU:
" File "...\phi3v.py", line 140, in run(args)
File "...\phi3v.py", line 103, in run generator.generate_next_token()
RuntimeError: Must call ComputeLogits before GenerateNextToken"

and

RuntimeError: IsDone() can't be called in the middle of processing logits

To Reproduce
Any attempt for inference

The code below works for me from Line 103 below:
generated_text = ""

    # Loop to generate each token after calling compute_logits  
    while True:  
        generator.compute_logits()  # Prepare the output logits  
        generator.generate_next_token()  # Generate the next token  
        new_tokens = generator.get_next_tokens()  # Get the output tokens  

        for token in new_tokens:  
            generated_text += tokenizer_stream.decode(token)  # Decode each token and accumulate text  

        if generator.is_done():  
            break  

    # Print out the generated tokens  
    print(generated_text)  

Desktop (please complete the following information):

• OS Win 11
• Python 3.12.5

[xunuohope1107]

Also facing the same issue.
root@993565088b12:/workspace# python3 phi3v.py -m cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4 -p cpu Loading model... Model loaded Image Path (comma separated; leave empty if no image): /workspace/coffee.png Using image: /workspace/coffee.png Prompt: what's in image? Processing images and prompt... Generating response... Traceback (most recent call last): File "/workspace/phi3v.py", line 140, in <module> run(args) File "/workspace/phi3v.py", line 103, in run generator.generate_next_token() RuntimeError: Must call ComputeLogits before GenerateNextToken

[xunuohope1107]

Also facing the same issue when use phi-3-vision. root@993565088b12:/workspace# python3 phi3v.py -m cpu_and_mobile/cpu-int4-rtn-block-32-acc-level-4 -p cpu Loading model... Model loaded Image Path (comma separated; leave empty if no image): /workspace/coffee.png Using image: /workspace/coffee.png Prompt: what's in image? Processing images and prompt... Generating response... Traceback (most recent call last): File "/workspace/phi3v.py", line 140, in <module> run(args) File "/workspace/phi3v.py", line 103, in run generator.generate_next_token() RuntimeError: Must call ComputeLogits before GenerateNextToken

Got the phi3v.py from curl https://raw.githubusercontent.com/microsoft/onnxruntime-genai/main/examples/python/phi3v.py -o phi3v.py

[kunal-vaishnavi]

You can see this announcement for more information on which example script to use.