Skip to content

Commit

Permalink
assistant: Use GPT 4 tokenizer for o3-mini (#24068)
Browse files Browse the repository at this point in the history
Sorry to dump an unsolicited PR for a hot feature! I'm sure someone else
was taking a look at this.

I noticed that token counting was disabled and I was getting error logs
of the form `[2025-01-31T22:59:01-05:00 ERROR assistant_context_editor]
No tokenizer found for model o3-mini` when using the new model. To fix
the issue, this PR registers the `gpt-4` tokenizer for this model.

Release Notes:

- openai: Fixed Assistant token counts for `o3-mini` models
  • Loading branch information
rpadaki authored Feb 1, 2025
1 parent f6824e3 commit af461f8
Showing 1 changed file with 4 additions and 1 deletion.
5 changes: 4 additions & 1 deletion crates/language_models/src/provider/open_ai.rs
Original file line number Diff line number Diff line change
Expand Up @@ -361,7 +361,10 @@ pub fn count_open_ai_tokens(
.collect::<Vec<_>>();

match model {
open_ai::Model::Custom { .. } | open_ai::Model::O1Mini | open_ai::Model::O1 => {
open_ai::Model::Custom { .. }
| open_ai::Model::O1Mini
| open_ai::Model::O1
| open_ai::Model::O3Mini => {
tiktoken_rs::num_tokens_from_messages("gpt-4", &messages)
}
_ => tiktoken_rs::num_tokens_from_messages(model.id(), &messages),
Expand Down

0 comments on commit af461f8

Please sign in to comment.