Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Pooling Strategy Option for embedding creation #491

Merged
merged 8 commits into from
Feb 6, 2025
Merged

Add Pooling Strategy Option for embedding creation #491

merged 8 commits into from
Feb 6, 2025

Conversation

VibhuJawa
Copy link
Collaborator

This PR closes: #490

@VibhuJawa
Add pooling stratedgy
b0e3317 
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
@VibhuJawa VibhuJawa self-assigned this Jan 20, 2025
@VibhuJawa
Ensure pytest is importable in a CPU only environment
bc83181 
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
@VibhuJawa VibhuJawa added the gpuci Run GPU CI/CD on PR label Jan 20, 2025
@VibhuJawa
Fix last token based on Avinash's feedback
e5d40ea 
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
@VibhuJawa VibhuJawa added gpuci Run GPU CI/CD on PR and removed gpuci Run GPU CI/CD on PR labels Jan 21, 2025
@VibhuJawa
Copy link
Collaborator Author

CC: @avem-nv to review

@VibhuJawa
Fix indexing issues
ac7a7fc 
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
@VibhuJawa
Merge in main
3cb6354 
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
@VibhuJawa
Merge in main
d876faa 
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
@VibhuJawa VibhuJawa added gpuci Run GPU CI/CD on PR and removed gpuci Run GPU CI/CD on PR labels Feb 5, 2025
@VibhuJawa VibhuJawa requested a review from sarahyurick February 5, 2025 19:29
@VibhuJawa
Fix Doc-string
b9ce56e 
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
@VibhuJawa VibhuJawa marked this pull request as ready for review February 5, 2025 19:44
sarahyurick
sarahyurick reviewed Feb 6, 2025
View reviewed changes
Copy link
Collaborator

@sarahyurick sarahyurick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

nemo_curator/modules/semantic_dedup/embeddings.py Outdated Show resolved Hide resolved
tests/test_semdedup.py Outdated Show resolved Hide resolved
@VibhuJawa
Address Sarah's reviews
04f8d55 
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
@sarahyurick sarahyurick added gpuci Run GPU CI/CD on PR and removed gpuci Run GPU CI/CD on PR labels Feb 6, 2025
@VibhuJawa VibhuJawa merged commit 97aa372 into NVIDIA:main Feb 6, 2025
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sarahyurick sarahyurick sarahyurick approved these changes

Assignees

@VibhuJawa VibhuJawa

Labels
gpuci Run GPU CI/CD on PR
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEA] Add option to extract last token embedding
2 participants
@VibhuJawa @sarahyurick