Mistral nits #57
Mistral nits #57
Conversation
It is required for Mistral models.
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| @classmethod | ||
| def from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs): | ||
| # Unless specified otherwise, the model weights type will be bfloat16 | ||
| torch_dtype = kwargs.pop("torch_dtype", torch.bfloat16) |
There was a problem hiding this comment.
Should we force torch.bfloat16 or rely on the config.json and use torch_dtype="auto" to respect the original checkpoint format?
There was a problem hiding this comment.
I think we should force to bfloat16 here, because that is what works faster on these TPUs
There was a problem hiding this comment.
I'm fine with forcing bfloat16 for the compilation phase, but I'm not so sure about from_pretrained ? Wouldnt a user get the exact same model from the hub and then do whatever he wants with it?
And if going to compile, use bfloat16? Your call if you think this is the way to go, but just maybe print something to say the dtype was changed somehow to inform the end user in this case
There was a problem hiding this comment.
If you use from_pretrained, the model ends up being loaded in float32, that is the default for transformers, unless something else is specified, and disregarding the config (that says how the model was trained if I got it right). What I am doing here then is changing the default dtype, I think it should OK and that is aligned with the torch_dtype in Mistral.
There was a problem hiding this comment.
I ended up adding the warning as suggested.
sentencepiecethat is making the nightly CI fail.bfloat16by default.