Painting pixels...why?

[jdeepwell]

When using Anthropic as well as Ollama, Witsy often tries to answer questions first by creating an image ("Painting pixels") – which is mostly useless (unless I explicitly ask for an image).

When I use Claude and Ollama directly (not via Witsy) they don't do that.

Is there a setting somewhere that I am missing?

[jdeepwell]

I also just saw that in the text response it says:

"Based on the tool call response, it seems that you are using a visual interface to input your question. However, I'll provide a text-based answer as requested."

[nbonamy]

By default, Witsy exposes all plugins to the model. Then the model makes its own decision to use them or not. What model and prompt are you using them you see this behavior. Note that you can disable plugins for a given conversation if you see a model particularly bad at handling tool presence:

[jdeepwell]

It mainly happens with Claude 3.5 Sonnet (New).

Prompt: "Explain the process of a TLS client handshake"

Turning it off on a per model basis would be great (I personally normally do not use Claude for image generation)…

Also: The hamburger button to disable the plugins only appears after the first response.

[nbonamy]

In the next version, a new menu icon will allow you to configure this before prompting. Thinking of adding a "save as default for this model" but it is a bit tricky...

https://github.com/nbonamy/witsy/releases/tag/v1.38.0

[jdeepwell]

Great 👍

The "default for this model" thing might be sensible as in general I think people tend to use certain models for certain tasks – so if its doable it would be quite nice IMHO.

For the original issue it would be more like a workaround than an actual solution, though – the actual question should probably be why the model thinks that for a prompt like "Explain the process of a TLS handshake" it is a good idea to create an image in the first place. Especially if the image it then generates is this:

🤣

I do not know enough on how plugins are exposes to the models and how they then decide on when/if to use which, but maybe there is some knob that Witsy could tune differently?

[nbonamy]

Tools are exposed part of the API call to the models. Witsy (as every other software) provides a list of tools and importantly a tool_choice flag which indicates how the model should use tools. Common values are auto (LLM decides), any (force LLM to use one of the tools (of its choice)) ,tool (force LLM to use a specific tool). Witsy uses auto for all providers (https://github.com/nbonamy/multi-llm-ts/blob/5cb3d5bedc2c232aaa298a29cea6a33f026eaad3/src/providers/anthropic.ts#L194) (https://github.com/nbonamy/multi-llm-ts/blob/main/src/providers/ollama.ts#L174).