Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert "perf(udp): multi-message receive on apple (#2302)" #2352

Closed
wants to merge 1 commit into from

Conversation

mxinden
Copy link
Collaborator

@mxinden mxinden commented Jan 13, 2025

This reverts commit 3c48567.

RPS benchmark sometimes fails with TransportError(PeerError(5). Git bisect shows 3c48567 introduced the regression.

Until I figure out the reason, let's revert.

Copy link

codecov bot commented Jan 13, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.30%. Comparing base (085fa62) to head (a8ebd40).
Report is 16 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2352      +/-   ##
==========================================
+ Coverage   95.29%   95.30%   +0.01%     
==========================================
  Files         114      114              
  Lines       36844    36858      +14     
  Branches    36844    36858      +14     
==========================================
+ Hits        35110    35128      +18     
+ Misses       1728     1724       -4     
  Partials        6        6              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

Failed Interop Tests

QUIC Interop Runner, client vs. server, differences relative to 085fa62.

neqo-latest as client

neqo-latest as server

All results

Succeeded Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

neqo-latest as server

Unsupported Interop Tests

QUIC Interop Runner, client vs. server

neqo-latest as client

neqo-latest as server

Copy link

Benchmark results

Performance differences relative to 085fa62.

decode 4096 bytes, mask ff: No change in performance detected.
       time:   [11.176 µs 11.223 µs 11.282 µs]
       change: [-0.0450% +0.6787% +1.4128%] (p = 0.08 > 0.05)

Found 21 outliers among 100 measurements (21.00%)
3 (3.00%) low severe
2 (2.00%) low mild
3 (3.00%) high mild
13 (13.00%) high severe

decode 1048576 bytes, mask ff: No change in performance detected.
       time:   [3.0161 ms 3.0258 ms 3.0369 ms]
       change: [-1.0595% -0.3860% +0.2125%] (p = 0.26 > 0.05)

Found 9 outliers among 100 measurements (9.00%)
1 (1.00%) low mild
8 (8.00%) high severe

decode 4096 bytes, mask 7f: No change in performance detected.
       time:   [19.534 µs 19.587 µs 19.644 µs]
       change: [-0.3124% +0.0163% +0.3351%] (p = 0.93 > 0.05)

Found 22 outliers among 100 measurements (22.00%)
1 (1.00%) low severe
5 (5.00%) low mild
1 (1.00%) high mild
15 (15.00%) high severe

decode 1048576 bytes, mask 7f: No change in performance detected.
       time:   [5.1607 ms 5.1723 ms 5.1854 ms]
       change: [-1.3430% -0.4242% +0.2013%] (p = 0.35 > 0.05)

Found 14 outliers among 100 measurements (14.00%)
14 (14.00%) high severe

decode 4096 bytes, mask 3f: No change in performance detected.
       time:   [5.5291 µs 5.5529 µs 5.5830 µs]
       change: [-0.5999% +1.5931% +5.3857%] (p = 0.59 > 0.05)

Found 11 outliers among 100 measurements (11.00%)
5 (5.00%) low mild
1 (1.00%) high mild
5 (5.00%) high severe

decode 1048576 bytes, mask 3f: No change in performance detected.
       time:   [1.7580 ms 1.7596 ms 1.7625 ms]
       change: [-0.0153% +0.0744% +0.2459%] (p = 0.52 > 0.05)

Found 6 outliers among 100 measurements (6.00%)
3 (3.00%) high mild
3 (3.00%) high severe

coalesce_acked_from_zero 1+1 entries: No change in performance detected.
       time:   [99.329 ns 99.656 ns 99.989 ns]
       change: [-0.0878% +0.3087% +0.7104%] (p = 0.13 > 0.05)

Found 13 outliers among 100 measurements (13.00%)
10 (10.00%) high mild
3 (3.00%) high severe

coalesce_acked_from_zero 3+1 entries: No change in performance detected.
       time:   [117.27 ns 117.69 ns 118.17 ns]
       change: [-0.1934% +0.3516% +0.8787%] (p = 0.22 > 0.05)

Found 17 outliers among 100 measurements (17.00%)
1 (1.00%) low mild
16 (16.00%) high severe

coalesce_acked_from_zero 10+1 entries: No change in performance detected.
       time:   [116.58 ns 116.91 ns 117.34 ns]
       change: [-0.2761% +0.2147% +0.7272%] (p = 0.42 > 0.05)

Found 15 outliers among 100 measurements (15.00%)
5 (5.00%) low severe
2 (2.00%) low mild
3 (3.00%) high mild
5 (5.00%) high severe

coalesce_acked_from_zero 1000+1 entries: No change in performance detected.
       time:   [97.317 ns 97.433 ns 97.564 ns]
       change: [-1.0142% -0.1978% +0.5591%] (p = 0.65 > 0.05)

Found 13 outliers among 100 measurements (13.00%)
7 (7.00%) high mild
6 (6.00%) high severe

RxStreamOrderer::inbound_frame(): Change within noise threshold.
       time:   [111.04 ms 111.10 ms 111.16 ms]
       change: [-0.3343% -0.1769% -0.0504%] (p = 0.01 < 0.05)

Found 16 outliers among 100 measurements (16.00%)
10 (10.00%) low mild
6 (6.00%) high mild

SentPackets::take_ranges: No change in performance detected.
       time:   [5.5160 µs 5.6975 µs 5.8932 µs]
       change: [-1.9785% +0.7002% +3.5085%] (p = 0.64 > 0.05)

Found 8 outliers among 100 measurements (8.00%)
8 (8.00%) high mild

transfer/pacing-false/varying-seeds: Change within noise threshold.
       time:   [41.575 ms 41.668 ms 41.772 ms]
       change: [-0.7369% -0.4437% -0.1270%] (p = 0.00 < 0.05)

Found 3 outliers among 100 measurements (3.00%)
1 (1.00%) low mild
1 (1.00%) high mild
1 (1.00%) high severe

transfer/pacing-true/varying-seeds: No change in performance detected.
       time:   [42.042 ms 42.123 ms 42.212 ms]
       change: [-0.1417% +0.1567% +0.4511%] (p = 0.30 > 0.05)

Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) low mild
1 (1.00%) high severe

transfer/pacing-false/same-seed: Change within noise threshold.
       time:   [41.552 ms 41.628 ms 41.712 ms]
       change: [-1.1799% -0.9313% -0.6763%] (p = 0.00 < 0.05)

Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high severe

transfer/pacing-true/same-seed: Change within noise threshold.
       time:   [41.659 ms 41.726 ms 41.801 ms]
       change: [+0.3295% +0.5793% +0.8316%] (p = 0.00 < 0.05)

Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high severe

1-conn/1-100mb-resp/mtu-1504 (aka. Download)/client: 💚 Performance has improved.
       time:   [891.10 ms 899.58 ms 908.39 ms]
       thrpt:  [110.08 MiB/s 111.16 MiB/s 112.22 MiB/s]
change:
       time:   [-4.2354% -2.8237% -1.2988%] (p = 0.00 < 0.05)
       thrpt:  [+1.3159% +2.9057% +4.4227%]
1-conn/10_000-parallel-1b-resp/mtu-1504 (aka. RPS)/client: No change in performance detected.
       time:   [314.87 ms 316.53 ms 318.19 ms]
       thrpt:  [31.428 Kelem/s 31.593 Kelem/s 31.759 Kelem/s]
change:
       time:   [-1.5746% -0.7046% +0.1574%] (p = 0.11 > 0.05)
       thrpt:  [-0.1571% +0.7096% +1.5998%]

Found 3 outliers among 100 measurements (3.00%)
2 (2.00%) low mild
1 (1.00%) high mild

1-conn/1-1b-resp/mtu-1504 (aka. HPS)/client: No change in performance detected.
       time:   [33.933 ms 34.142 ms 34.372 ms]
       thrpt:  [29.093  elem/s 29.289  elem/s 29.470  elem/s]
change:
       time:   [-1.3486% -0.4978% +0.2746%] (p = 0.24 > 0.05)
       thrpt:  [-0.2739% +0.5003% +1.3670%]

Found 8 outliers among 100 measurements (8.00%)
1 (1.00%) low mild
4 (4.00%) high mild
3 (3.00%) high severe

1-conn/1-100mb-resp/mtu-1504 (aka. Upload)/client: Change within noise threshold.
       time:   [1.6633 s 1.6825 s 1.7014 s]
       thrpt:  [58.774 MiB/s 59.435 MiB/s 60.122 MiB/s]
change:
       time:   [-3.1501% -1.5471% -0.1051%] (p = 0.04 < 0.05)
       thrpt:  [+0.1052% +1.5714% +3.2526%]

Client/server transfer results

Transfer of 33554432 bytes over loopback.

Client Server CC Pacing MTU Mean [ms] Min [ms] Max [ms]
gquiche gquiche 1504 552.6 ± 67.6 503.9 687.3
neqo gquiche reno on 1504 747.7 ± 56.7 702.0 903.4
neqo gquiche reno 1504 743.3 ± 16.8 715.0 771.1
neqo gquiche cubic on 1504 779.9 ± 45.3 753.5 906.5
neqo gquiche cubic 1504 747.3 ± 24.6 701.2 788.1
msquic msquic 1504 165.6 ± 86.8 102.5 356.8
neqo msquic reno on 1504 270.6 ± 78.9 213.2 471.9
neqo msquic reno 1504 272.9 ± 87.9 206.3 439.4
neqo msquic cubic on 1504 275.4 ± 123.2 210.4 644.8
neqo msquic cubic 1504 273.7 ± 107.1 204.5 536.1
gquiche neqo reno on 1504 719.0 ± 85.7 607.7 882.4
gquiche neqo reno 1504 680.9 ± 86.8 559.0 808.9
gquiche neqo cubic on 1504 685.0 ± 106.1 556.1 912.1
gquiche neqo cubic 1504 677.8 ± 80.9 565.6 801.2
msquic neqo reno on 1504 515.5 ± 62.1 481.7 691.1
msquic neqo reno 1504 485.8 ± 42.4 458.5 604.7
msquic neqo cubic on 1504 460.7 ± 9.5 448.7 476.2
msquic neqo cubic 1504 465.2 ± 10.9 450.2 486.5
neqo neqo reno on 1504 463.0 ± 53.5 426.8 608.3
neqo neqo reno 1504 436.1 ± 13.1 407.7 452.1
neqo neqo cubic on 1504 527.4 ± 73.7 468.7 682.5
neqo neqo cubic 1504 448.7 ± 6.5 439.8 458.2

⬇️ Download logs

@larseggert
Copy link
Collaborator

I wonder if I messed something up on the quinn-udp side. Off-by-one error on TX or RX when dealing with the buffer vector? Hm.

@mxinden
Copy link
Collaborator Author

mxinden commented Jan 14, 2025

@larseggert but that would not explain why it is failing for me on Linux, right?

@mxinden
Copy link
Collaborator Author

mxinden commented Jan 14, 2025

Debugging this further, the TransportError(PeerError(5)) is originating here:

if ss.is_none() && rs.is_none() && !existed {
return Err(Error::StreamStateError);
}

@mxinden
Copy link
Collaborator Author

mxinden commented Jan 14, 2025

Ah, I believe existed is wrong:

let existed = !stream_id.is_remote_initiated(self.role)
&& self.local_stream_limits[stream_id.stream_type()].used() > stream_id.index();
let ss = self.send.get_mut(stream_id).ok();
let rs = self.recv.get_mut(stream_id).ok();
if ss.is_none() && rs.is_none() && !existed {
return Err(Error::StreamStateError);
}

Say that the stream is remote initiated, then existed is false and then ss.is_none() && rs.is_none() && !existed is true.

I will create a pull request.

Edit: #2358

@mxinden
Copy link
Collaborator Author

mxinden commented Jan 14, 2025

Closing here in favor of #2358.

@mxinden mxinden closed this Jan 14, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants