[JAX][Common] Support GQA #578
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/te-ci |
zlsh80826
force-pushed
the
rewang/gqa-clean
branch
2 times, most recently
from
0f641e6 to
000548c
Compare
|
/te-ci jax |
cyanguwa
reviewed
Jan 5, 2024
transformer_engine/common/fused_attn/fused_attn_f16_arbitrary_seqlen.cu
Outdated
Show resolved
Hide resolved
zlsh80826
force-pushed
the
rewang/gqa-clean
branch
from
000548c to
5372c5c
Compare
|
/te-ci |
zlsh80826
force-pushed
the
rewang/gqa-clean
branch
from
26fa707 to
13b4cc0
Compare
|
/te-ci |
1 similar comment
|
/te-ci |
zlsh80826
force-pushed
the
rewang/gqa-clean
branch
from
b538b4a to
13b4cc0
Compare
|
/te-ci |
Signed-off-by: Reese Wang <rewang@nvidia.com>
Signed-off-by: Reese Wang <rewang@nvidia.com>
Signed-off-by: Reese Wang <rewang@nvidia.com>
Signed-off-by: Reese Wang <rewang@nvidia.com>
Signed-off-by: Reese Wang <rewang@nvidia.com>
Signed-off-by: Reese Wang <rewang@nvidia.com>
Signed-off-by: Reese Wang <rewang@nvidia.com>
Signed-off-by: Reese Wang <rewang@nvidia.com>
Signed-off-by: Reese Wang <rewang@nvidia.com>
Signed-off-by: Reese Wang <rewang@nvidia.com>
Signed-off-by: Reese Wang <rewang@nvidia.com>
Signed-off-by: Reese Wang <rewang@nvidia.com>
Signed-off-by: Reese Wang <rewang@nvidia.com>
zlsh80826
force-pushed
the
rewang/gqa-clean
branch
from
acc308a to
bbe7066
Compare
|
/te-ci |
Signed-off-by: Reese Wang <rewang@nvidia.com>
|
/te-ci |
|
@cyanguwa @denera @mingxu1067, all unit tests passed. Could you help review again? Thanks |
Wong4j
pushed a commit
to Wong4j/TransformerEngine
that referenced
this pull request
Jan 22, 2024
* Support num_gqa_groups arguments Signed-off-by: Reese Wang <rewang@nvidia.com> * Add GQA support on the JAX bridge code Signed-off-by: Reese Wang <rewang@nvidia.com> * Fix the kv stride of the arbitrary backend Signed-off-by: Reese Wang <rewang@nvidia.com> * Complete rewrite fused attention tests and add GQA coverage Signed-off-by: Reese Wang <rewang@nvidia.com> * Support unfused GQA Signed-off-by: Reese Wang <rewang@nvidia.com> * Calculate seqlen before the primitive for the better perf Signed-off-by: Reese Wang <rewang@nvidia.com> * Add GQA layer tests Signed-off-by: Reese Wang <rewang@nvidia.com> * Apply code style checks for te_jax Signed-off-by: Reese Wang <rewang@nvidia.com> * Apply code style checks for tests Signed-off-by: Reese Wang <rewang@nvidia.com> * Add num_gqa_groups doc Signed-off-by: Reese Wang <rewang@nvidia.com> * Refine the qkv_type Signed-off-by: Reese Wang <rewang@nvidia.com> * Correct the variable naming Signed-off-by: Reese Wang <rewang@nvidia.com> * Handle Max512 CAUSAL Signed-off-by: Reese Wang <rewang@nvidia.com> * Add WAR for the latest jax image Signed-off-by: Reese Wang <rewang@nvidia.com> --------- Signed-off-by: Reese Wang <rewang@nvidia.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
num_gqa_groups) for both fused attention and unfused attention implementation.kv_strideof the flash attention