Skip to content

Commit

Permalink
Update TorchBench commit to main (#145455)
Browse files Browse the repository at this point in the history
Summary:
I'm adding sam2 to TorchBench #2566, so, as part of that, I'm updating PyTorch CI to use latest TorchBench commit.

The corresponding change from TorchBench is #2584

The main thing to call out that the newer transformers added by #2488 is regressing several models. This needs to be investigated further, and I pin the version to unblock this change.

* `hf_Roberta_base` a new model added by #2279, not sure why it fails accuracy on A10G, but it works fine on A100
* `speech_transformer` failures are pre-existing trunk failures, i.e. https://github.com/pytorch/pytorch/actions/runs/13040114684/job/36380989702#step:22:2408

X-link: pytorch/pytorch#145455
Approved by: https://github.com/kit1980

Reviewed By: ZainRizvi

Differential Revision: D69056903

fbshipit-source-id: 6b98c246b9dc8811257ec46d4821ba6ee3363dfe
  • Loading branch information
huydhn authored and facebook-github-bot committed Feb 3, 2025
1 parent 3e5a389 commit 76d4920
Show file tree
Hide file tree
Showing 2 changed files with 3 additions and 2 deletions.
2 changes: 1 addition & 1 deletion userbenchmark/dynamo/dynamobench/common.py
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ class CI(NamedTuple):
"stable_diffusion_text_encoder",
"timm_efficientdet",
"timm_nfnet",
"timm_regnet",
"timm_resnest",
"timm_vision_transformer",
"timm_vovnet",
"vgg16",
Expand Down
3 changes: 2 additions & 1 deletion userbenchmark/dynamo/dynamobench/torchbench.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -34,14 +34,14 @@ tolerance:
- vgg16
- mobilenet_v3_large
- nvidia_deeprecommender
- timm_efficientdet

# These models need >1e-3 tolerance
even_higher:
- soft_actor_critic
- tacotron2
- yolov3
- timm_efficientdet
- timm_efficientnet
- squeezenet1_1

higher_fp16:
Expand All @@ -66,6 +66,7 @@ tolerance:

require_larger_multiplier_for_smaller_tensor:
- yolov3
- timm_efficientnet

# These benchmarks took >600s on an i9-11900K CPU
very_slow: &VERY_SLOW_MODELS
Expand Down

0 comments on commit 76d4920

Please sign in to comment.