Prefer chat_template.json for chat template

[DePasqualeOrg]

ml-explore/mlx-swift-examples#197 (comment)

[FL33TW00D]

Looks good to me @pcuenca

[pcuenca]

Taking a look now

[pcuenca]

Thanks for another great contribution @DePasqualeOrg!

There are a couple of subtleties about chat templates that I'm willing to ignore (unless I'm missing something important), and some minor code changes that I think could make the code a bit cleaner – although this could admittedly be subjective.

[pcuenca]

Not a fan of repeating this code block and the return here, inside the nested ifs.

I'd recommend we write a helper function to potentially update the chat template, such as:

    func updatedTokenizerConfig(tokenizerConfig: Config?, chatTemplateConfig: Config?) -> Config? {
        guard
            let chatTemplateConfig = chatTemplateConfig,
            let overrideChatTemplate = chatTemplateConfig.chatTemplate?.stringValue else {
            return tokenizerConfig
        }

        var configDict = tokenizerConfig?.dictionary ?? [:]
        configDict["chat_template"] = overrideChatTemplate
        return Config(configDict)
    }

And then we can just use this before the return:

        let configs = Configurations(
            modelConfig: modelConfig,
            tokenizerConfig: updatedTokenizerConfig(tokenizerConfig: tokenizerConfig, chatTemplateConfig: chatTemplateConfig),
            tokenizerData: tokenizerVocab
        )

[pcuenca]

Suggested change

	XCTAssertTrue(qwen2VLEncoded == qwen2_5VLEncoded)
	XCTAssertTrue(qwen2VLDecoded == qwen2_5VLDecoded && qwen2_5VLDecoded == expectedOutput)
	XCTAssertEqual(qwen2VLEncoded, qwen2_5VLEncoded, "Encoded sequences should be equal")
	XCTAssertEqual(qwen2VLDecoded, qwen2_5VLDecoded, "Decoded sequences should be equal")
	XCTAssertEqual(qwen2_5VLDecoded, expectedOutput, "Decoded should match expected")

nit, should provide better error messages maybe

[pcuenca]

Technically, this is not the same algorithm used in transformers. IIRC, if we instantiate a tokenizer from a repo where the tokenizer has a chat template and a different chat_template.json, the template from the tokenizer will still be used. However, if we instantiate a processor, then the chat_template.json will be used.

I'm willing to diverge from this behaviour, given that:

• Chat template divergence should not be expected (it's possibly a mistake if both templates differ)
• Processor and tokenizer templates should be synced at some point.
• There is no processor abstraction in swift-transformers.

cc @Rocketknight1 in case I'm missing some weird edge case.

[pcuenca]

Technically, this could potentially be an array too. But this is discouraged.

[DePasqualeOrg]

Thanks for your suggestions, @pcuenca. I made some changes based on your input.

[DePasqualeOrg]

We'll need a new version tag in this repo to pick this up in mlx-swift-examples so that we can make Qwen 2.5 VL work with the correct chat template.

[pcuenca]

Added