Experimental: tokenizers with and without templates

[pcuenca]

This builds on top of #166 by @greenrazer.

The goal is to be able to opt-in to the chat template feature, which carries the Jinja dependency, which in turn depends on swift-collections and whatnot. It works in its current state, but it's ugly – I found way more cross-module quirks than I was expecting. I wanted to make it as easy as possible for consumers and allow to use either version (with or without templates) from their SPM manifests.

Opinions, corrections, and alternative ideas are encouraged and most welcome!

---

How to use:

• If you want to use the core Tokenizers functionality, without the chat templates:

Add the following dependency to your target, as usual (except Tokenizers is now a product, no need to use the full Transformers lib). This applies to projects such as WhisperKit.

            dependencies: [
                .product(name: "Tokenizers", package: "swift-transformers"),
            ],

• To opt-in to using chat-templates:

This applies to projects such as mlx-swift-examples.

            dependencies: [
                .product(name: "Tokenizers", package: "swift-transformers"),
+               .product(name: "TokenizersTemplates", package: "swift-transformers"),
            ],

That's it, you don't need to import TokenizersTemplates or do anything other than just declaring it as a dependency.

---

Known issues:

• I haven't looked at tests yet, they probably won't compile.
• I know there are conflicts because of last night's tools PR. Let's agree on the general direction before addressing them.

[pcuenca]

See comment in TokenizersTemplates

[pcuenca]

Yes, if these overrides don't exist, the linker can't find the implementations.

[pcuenca]

This is a good chunk of the magic. The new Tokenizers implementation is just this wrapper file, which exposes the imported TokenizersCore types.

[pcuenca]

So if TokenizersTemplates is available (because users have declared it as a dependency), we override the definition so the factory below uses the subclass.

[pcuenca]

This could be moved back to TokenizersCore, but then we'd need a typealias here as well (and a subclass).

[pcuenca]

cc @greenrazer @FL33TW00D @Vaibhavs10 for opinions and feedback.

[greenrazer]

I like this solution. however, I cannot get the canImport approach in TokenizersWrapper to work for the tests due to how Swift Package Manager handles module dependencies and conditional imports within the same package. All the tests work great with almost no modification (besides changing Tokenizers to TokenizersCore) except for the ChatTemplatesTests.swift.

I've tried a few different options:

• The best option: Just adding TokenizersTemplates as a dependency to the test, similar to how you would for an external package. This doesn’t work because it doesn’t trigger recompilation for the test.
• Adding swiftSettings: [.define("USE_TEMPLATES")] to the test and then adding another build condition to TokenizersWrapper. Same issue as above—it doesn’t trigger recompilation for the test.
• Creating two targets pointing to the same TokenizersWrapper source file, with one including USE_TEMPLATES. This doesn’t work because you can't have two targets referencing the same source file.
• Adding .target(name: "TokenizersTemplates", condition: .when(platforms: nil)) as a dependency to hopefully run only during testing. This doesn’t work in newer Swift versions.
• Adding swiftSettings: [.define("ENABLE_TEMPLATES", .when(configuration: .debug))] to the Tokenizers target. This almost works, but TokenizersTemplates must then be a dependency of Tokenizers, which defeats the whole purpose.

Since users must explicitly import TokenizersTemplates anyway, I think the best approach is to create two wrappers and two targets. That way, users only need to import either TokenizersTemplates or Tokenizers, depending on their needs.