Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: Don't fail the bench job if the cache is missing #2437

Merged
merged 1 commit into from
Feb 11, 2025

Conversation

larseggert
Copy link
Collaborator

Thanks you GitHub for wiping our cache so we could debug this issue.

Thanks you GitHub for wiping our cache so we could debug this issue.
Copy link

codecov bot commented Feb 11, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.28%. Comparing base (1d9de16) to head (39fa41f).
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2437      +/-   ##
==========================================
- Coverage   95.28%   95.28%   -0.01%     
==========================================
  Files         114      114              
  Lines       37111    37111              
  Branches    37111    37111              
==========================================
- Hits        35363    35362       -1     
- Misses       1742     1743       +1     
  Partials        6        6              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

Failed Interop Tests

QUIC Interop Runner, client vs. server, differences relative to cb058cb.

neqo-latest as client

neqo-latest as server

All results

Succeeded Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

neqo-latest as server

Unsupported Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

neqo-latest as server

Copy link

Benchmark results

decode 4096 bytes, mask ff:
       time:   [10.865 µs 10.905 µs 10.949 µs]
Found 18 outliers among 100 measurements (18.00%)
  2 (2.00%) low severe
  5 (5.00%) low mild
  1 (1.00%) high mild
  10 (10.00%) high severe
decode 1048576 bytes, mask ff:
       time:   [3.1291 ms 3.1392 ms 3.1509 ms]
Found 10 outliers among 100 measurements (10.00%)
  10 (10.00%) high severe
decode 4096 bytes, mask 7f:
       time:   [17.645 µs 17.696 µs 17.752 µs]
Found 23 outliers among 100 measurements (23.00%)
  2 (2.00%) low severe
  4 (4.00%) low mild
  2 (2.00%) high mild
  15 (15.00%) high severe
decode 1048576 bytes, mask 7f:
       time:   [5.4028 ms 5.4143 ms 5.4273 ms]
Found 14 outliers among 100 measurements (14.00%)
  14 (14.00%) high severe
decode 4096 bytes, mask 3f:
       time:   [6.6806 µs 6.7101 µs 6.7439 µs]
Found 9 outliers among 100 measurements (9.00%)
  3 (3.00%) low mild
  4 (4.00%) high mild
  2 (2.00%) high severe
decode 1048576 bytes, mask 3f:
       time:   [1.7585 ms 1.7640 ms 1.7711 ms]
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) low mild
  3 (3.00%) high severe
coalesce_acked_from_zero 1+1 entries:
       time:   [90.958 ns 91.267 ns 91.579 ns]
Found 9 outliers among 100 measurements (9.00%)
  7 (7.00%) high mild
  2 (2.00%) high severe
coalesce_acked_from_zero 3+1 entries:
       time:   [109.14 ns 109.45 ns 109.78 ns]
Found 18 outliers among 100 measurements (18.00%)
  2 (2.00%) low severe
  3 (3.00%) low mild
  7 (7.00%) high mild
  6 (6.00%) high severe
coalesce_acked_from_zero 10+1 entries:
       time:   [108.98 ns 109.49 ns 110.05 ns]
Found 10 outliers among 100 measurements (10.00%)
  1 (1.00%) low severe
  1 (1.00%) low mild
  3 (3.00%) high mild
  5 (5.00%) high severe
coalesce_acked_from_zero 1000+1 entries:
       time:   [93.029 ns 93.164 ns 93.314 ns]
Found 8 outliers among 100 measurements (8.00%)
  2 (2.00%) high mild
  6 (6.00%) high severe
RxStreamOrderer::inbound_frame():
       time:   [111.48 ms 111.54 ms 111.59 ms]
Found 22 outliers among 100 measurements (22.00%)
  3 (3.00%) low severe
  6 (6.00%) low mild
  13 (13.00%) high mild
SentPackets::take_ranges:
       time:   [5.2567 µs 5.4084 µs 5.5621 µs]
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe
transfer/pacing-false/varying-seeds:
       time:   [36.454 ms 36.519 ms 36.583 ms]
transfer/pacing-true/varying-seeds:
       time:   [36.470 ms 36.524 ms 36.578 ms]
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) low mild
transfer/pacing-false/same-seed:
       time:   [36.603 ms 36.664 ms 36.725 ms]
transfer/pacing-true/same-seed:
       time:   [36.484 ms 36.553 ms 36.625 ms]
1-conn/1-100mb-resp/mtu-1504 (aka. Download)/client:
       time:   [859.57 ms 868.27 ms 877.33 ms]
       thrpt:  [113.98 MiB/s 115.17 MiB/s 116.34 MiB/s]
1-conn/10_000-parallel-1b-resp/mtu-1504 (aka. RPS)/client:
       time:   [318.21 ms 321.85 ms 325.44 ms]
       thrpt:  [30.727 Kelem/s 31.070 Kelem/s 31.426 Kelem/s]
1-conn/1-1b-resp/mtu-1504 (aka. HPS)/client:
       time:   [25.350 ms 25.521 ms 25.698 ms]
       thrpt:  [38.914  elem/s 39.184  elem/s 39.448  elem/s]
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high mild
1-conn/1-100mb-resp/mtu-1504 (aka. Upload)/client:
       time:   [1.8299 s 1.8495 s 1.8690 s]
       thrpt:  [53.503 MiB/s 54.068 MiB/s 54.647 MiB/s]

Client/server transfer results

Transfer of 33554432 bytes over loopback, 30 runs. All unit-less numbers are in milliseconds.

Client Server CC Pacing Mean ± σ Min Max Δ main Δ main
neqo neqo reno on 548.5 ± 56.6 455.9 697.0
neqo neqo reno 563.2 ± 132.8 464.8 1232.0
neqo neqo cubic on 554.9 ± 48.8 475.7 713.2
neqo neqo cubic 551.3 ± 35.3 483.7 667.5
google neqo reno on 905.9 ± 106.8 654.8 1155.8
google neqo reno 903.2 ± 93.9 643.8 986.7
google neqo cubic on 897.3 ± 97.7 659.5 998.3
google neqo cubic 901.3 ± 108.9 649.4 1143.9
google google 544.9 ± 33.2 521.3 696.8
neqo msquic reno on 226.7 ± 32.1 206.1 377.1
neqo msquic reno 224.2 ± 19.2 201.3 276.8
neqo msquic cubic on 232.7 ± 50.7 200.7 447.3
neqo msquic cubic 227.2 ± 22.6 205.5 328.2
msquic msquic 114.3 ± 12.8 96.6 148.9

⬇️ Download logs

@larseggert larseggert merged commit 9734cf2 into mozilla:main Feb 11, 2025
72 of 75 checks passed
@larseggert larseggert deleted the ci-bench-handle-missing-cache branch February 12, 2025 09:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant