Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make_sampler creates sampler chain with all sampling parameters #1330

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

neilmehta24
Copy link
Contributor

Closes #1219

Apply samplers in this order: top-k --> top-p --> min-p --> temp

The sampler methods were refactored to return the transformed logits instead of a token. Now, we can can chain them together to apply a series of sampling methods.

The method names of the samplers were changed to indicate that the implementation has changed, i.e., each method returns logits instead of a token. I am not sure about the API stability requirements, so let me know if you have a preference on the method name.

The temperature adjustment was removed from each sampler application, and instead it's applied only once as the very last step.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[mlx-lm] Allow samplers to be chainable
1 participant