Is there any way to distinguish LLMs and VLMs

[ibrahimcetin]

Description

Hi, I am developing an app which can run LLMs and VLMs based on user selection but LLMFactory and VLMFactory are separated so I need to run the configuration on the correct factory.

Firstly, I noticed that ModelRegistry.configuration doesn't return null if the configuration is not in the registry. Also, I cannot create custom ModelRegistry instance because its init internal, there is only LLMModelFactory.shared.modelRegistry to use.

Currently I use this workaround:

let isLLM = LLMModelFactory.shared.modelRegistry.models.contains { $0.name == modelConfiguration.name }

if isLLM {
    await loadModel(factory: LLMModelFactory.shared)
} else {
    await loadModel(factory: VLMModelFactory.shared)
}

But returning nil from ModelRegistry.configuration can be more appropriate.

Proposed Solution

I think ModelRegistry.configuration should return nil if the configuration isn't in the registry. Also ModelRegistry init may be public.

What do you think about it? If these make sense I can create a PR.

[DePasqualeOrg]

In Local Chat, I have a model configuration that includes the input types that the model accepts, and if the model accepts images or video, I use a computed variable to determine whether to use LLMModelFactory or VLMModelFactory. It might make sense to do something like that here too.

[davidkoski]

You can also use is VLMModel or is LLMModel though typically this type of reflection is considered a bad code smell.

I think ModelRegistry.configuration should return nil if the configuration isn't in the registry.

I think configuration(id: ) should probably remain as-is as there may be other callers and the current behavior is certainly how the command line tool expects it to work. I think it would be fine to add a contains(id:) -> Bool or configurationIfPresent(id: ) -> ModelConfiguration

Also ModelRegistry init may be public.

Yes, good idea. Same for ProcessorTypeRegistry and VLMModelFactory, etc.

Perhaps the initialization of the registries should move into shared:

public class VLMModelFactory: ModelFactory {

    public static let shared = VLMModelFactory()

My intent was that you could use these default instances to get default behavior but app developers could create their own registries as needed (though clearly I didn't try it as you can't!)

What do you think about it? If these make sense I can create a PR.

Please do!