Model testing (ollama) - and some tips and tricks

[mamema]

i was trying to optimize the outcome of tagging and title assignment, so i was thinking what can i do and made some tests:

• even though, the solution and the example prompt is in english, results are better if you translate your prompt to the document language you're analyzing
• even though we're having a playground to test prompting, which is great, i had tremendous success to ask the "best machine" (openai) we have to optimize prompting for another machine (ollama). My thinking was: i'm a human and i haven't developed those machines, so i do not know what they really need in good prompting. Refining the outcome of my prompts with AI are measurable.

Additional to the default prompt i had three major requirements: Use the year of the document as tag. Don't use my name within tags or correspondents. Use my language for tags.
Your prompt may don't need those requirements, but for me this was a great way to differentiate the following list of models.

Every model got the same prompt to work with.

Mistral7B:

• good Tag selection
• but never assigned the Year as Tag, which is a requirement
• out of 14 documrnts 8 were not tagged at all
• generated tags called "tag1","tag2","tag3"
• used english tags even though forbidden

llama2:7b

• tag selection not as good as mistral
• also no year tag attached
• Correspondent MyName was used, even though forbidden in prompt
• Correspondent "xxxxxxxx" assigned. useless. Document is OCRed
• generated tags called "tag1","tag2","tag3"
• out of 14 documrnts 8 were not tagged at all

Mistral-NeMo

• good Tag selection
• only partly on some documrnts assigned the Year as Tag, which is a requirement
• out of 14 documrnts 8 were not tagged at all
• generated tags called "tag1","tag2","tag3"
• My Name assigned as tag, even though forbidden
• out of 14 documrnts 4 were not tagged at all

phi4

• constantly tagged the year
• good tagging
• tagged with My Name which isn't allowed
• good on medical term tagging
• out of 14 documents 6 were not tagged at all
• used english tag terms even though forbidden in the prompt

gemma2:9b

• year gets extracted
• tag quality is good
• 4 out of 14 documents are not tagged at all

For me, the winner is "gemma2:9b", your mileage may vary.