Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

#0: temp workaround on TG resnet trace+2cq hang #18750

Merged
merged 1 commit into from
Mar 8, 2025
Merged

#0: temp workaround on TG resnet trace+2cq hang #18750

merged 1 commit into from
Mar 8, 2025

Conversation

yugaoTT
Copy link
Contributor

@yugaoTT yugaoTT commented Mar 6, 2025 •
edited
Loading

P0 temp fix for resnet50, which is hanging ND on TG for trace+2cq, due to the extra rt arg being send.
IT removes the extra args for sharded case.
#18724 (comment)

Checklist

@yugaoTT yugaoTT self-assigned this Mar 6, 2025
@tt-rkim
Copy link
Collaborator

tt-rkim commented Mar 7, 2025 •
edited
Loading

Running single card full pipes to be sure: https://github.com/tenstorrent/tt-metal/actions/runs/13719964134

This will probably go in though

@yugaoTT yugaoTT mentioned this pull request Mar 7, 2025
Open
@tt-rkim
Copy link
Collaborator

tt-rkim commented Mar 7, 2025

Looks like slight regression: https://github.com/tenstorrent/tt-metal/actions/runs/13719964134/job/38373196371#step:10:65

We can bump this for now?

@yugaoTT
Copy link
Contributor Author

yugaoTT commented Mar 7, 2025

@tt-rkim is that pipeline stable ? my change shouldn't make it worse (should be slight better if there's any)

@yugaoTT
Copy link
Contributor Author

yugaoTT commented Mar 7, 2025

re-ran https://github.com/tenstorrent/tt-metal/actions/runs/13727917295

@tt-rkim tt-rkim requested review from esmalTT, uaydonat and a team as code owners March 7, 2025 19:35
@tt-rkim
Copy link
Collaborator

tt-rkim commented Mar 7, 2025

Yes that test is definitely stable...

@tt-rkim
Copy link
Collaborator

tt-rkim commented Mar 7, 2025

Oh wait - that wasn't a mistake
I will bump it down. We haven't seen that before

@tt-rkim
Copy link
Collaborator

tt-rkim commented Mar 7, 2025

cc: @esmalTT - bumping down experimental unet perf test threshold in e2e perf

@tt-rkim
Copy link
Collaborator

tt-rkim commented Mar 7, 2025

device perf: https://github.com/tenstorrent/tt-metal/actions/runs/13728005038
model perf: https://github.com/tenstorrent/tt-metal/actions/runs/13728027752

@yugaoTT @tt-rkim
#0: temp workaround on TG resnet trace+2cq hang
a78d90c 
#0: fixes

#0: Bump down unet fps to accommodate

Revert "#0: Bump down unet fps to accommodate"

This reverts commit 744a4ad.

Revert "Revert "#0: Bump down unet fps to accommodate""

This reverts commit e045504.
@tt-rkim tt-rkim merged commit 15db9cc into main Mar 8, 2025
14 checks passed
@tt-rkim tt-rkim deleted the yugao/resnet branch March 8, 2025 00:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@esmalTT esmalTT esmalTT approved these changes

@bbradelTT bbradelTT bbradelTT approved these changes

@TT-BrianLiu TT-BrianLiu Awaiting requested review from TT-BrianLiu TT-BrianLiu is a code owner

@vsureshTT vsureshTT Awaiting requested review from vsureshTT vsureshTT is a code owner

@edwinleeTT edwinleeTT Awaiting requested review from edwinleeTT edwinleeTT is a code owner

@uaydonat uaydonat Awaiting requested review from uaydonat uaydonat is a code owner

Assignees

@yugaoTT yugaoTT

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@yugaoTT @tt-rkim @esmalTT @bbradelTT