Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix_: nightly test runs #5791

Merged
merged 14 commits into from
Sep 3, 2024
Merged

Conversation

igor-sirotin
Copy link
Collaborator

@igor-sirotin igor-sirotin commented Aug 30, 2024

Fixes #5401

#5731 made nightly tests extremely long.

go test ./protocol -test.count=20 was run in a single process, so it was taking ~30min * 20runs = 10hours.
This is not acceptable. Quickly making the protocol package run faster didn't seem possible, so I reverted the solution to spawn multiple processes.

Yet, the result utilizes both approaches.
The protocol package is now an exclusion, all other packages are run with default go test behaviour.

Also fixed the coverage reporting for nightly tests

Test run for count=20: https://ci.status.im/job/status-go/job/tests-nightly/366/

NOTE: I currently included #5793 into this PR.
Will remove it when it's merged.

@igor-sirotin igor-sirotin marked this pull request as draft August 30, 2024 20:10
@status-im-auto
Copy link
Member

status-im-auto commented Aug 30, 2024

Jenkins Builds

Click to see older builds (25)
Commit #️⃣ Finished (UTC) Duration Platform Result
✔️ ed419b0 #1 2024-08-30 20:12:44 ~2 min tests-rpc 📄log
✔️ ed419b0 #1 2024-08-30 20:14:38 ~4 min linux 📦zip
✔️ ed419b0 #1 2024-08-30 20:14:57 ~4 min ios 📦zip
✔️ ed419b0 #1 2024-08-30 20:15:41 ~5 min android 📦aar
✔️ ed419b0 #1 2024-08-30 20:43:01 ~32 min tests 📄log
✔️ e93440e #2 2024-08-30 20:16:23 ~2 min tests-rpc 📄log
✔️ e93440e #2 2024-08-30 20:17:06 ~2 min linux 📦zip
✔️ e93440e #2 2024-08-30 20:17:35 ~1 min android 📦aar
✔️ e93440e #2 2024-08-30 20:18:08 ~2 min ios 📦zip
✔️ e93440e #2 2024-08-30 21:14:21 ~31 min tests 📄log
✔️ 0f059b8 #3 2024-08-31 00:04:03 ~1 min android 📦aar
✔️ 0f059b8 #3 2024-08-31 00:04:38 ~2 min linux 📦zip
✔️ 0f059b8 #3 2024-08-31 00:05:01 ~2 min tests-rpc 📄log
✔️ 0f059b8 #3 2024-08-31 00:05:49 ~3 min ios 📦zip
✔️ 0f059b8 #3 2024-08-31 00:34:39 ~32 min tests 📄log
✔️ 7919ec3 #4 2024-08-31 16:40:49 ~1 min android 📦aar
✔️ 7919ec3 #4 2024-08-31 16:41:35 ~2 min tests-rpc 📄log
✔️ 7919ec3 #4 2024-08-31 16:42:21 ~3 min ios 📦zip
✔️ 7919ec3 #4 2024-08-31 16:42:53 ~3 min linux 📦zip
✔️ 7919ec3 #4 2024-08-31 17:10:05 ~30 min tests 📄log
✔️ 6e9f57c #5 2024-09-02 13:53:26 ~1 min android 📦aar
✔️ 6e9f57c #5 2024-09-02 13:54:09 ~2 min linux 📦zip
✔️ 6e9f57c #5 2024-09-02 13:54:14 ~2 min tests-rpc 📄log
✔️ 6e9f57c #5 2024-09-02 13:55:07 ~3 min ios 📦zip
✔️ 6e9f57c #5 2024-09-02 14:23:18 ~31 min tests 📄log
Commit #️⃣ Finished (UTC) Duration Platform Result
✔️ f209106 #6 2024-09-02 22:09:19 ~2 min tests-rpc 📄log
✔️ f209106 #6 2024-09-02 22:09:22 ~2 min android 📦aar
✔️ f209106 #6 2024-09-02 22:09:34 ~2 min linux 📦zip
✔️ f209106 #6 2024-09-02 22:10:34 ~3 min ios 📦zip
✔️ f209106 #6 2024-09-02 22:38:14 ~31 min tests 📄log
✔️ e8d67cd #7 2024-09-03 10:51:37 ~2 min tests-rpc 📄log
✔️ e8d67cd #7 2024-09-03 10:51:55 ~2 min linux 📦zip
✔️ e8d67cd #7 2024-09-03 10:53:27 ~4 min ios 📦zip
✔️ e8d67cd #7 2024-09-03 10:54:49 ~5 min android 📦aar
✔️ e8d67cd #7 2024-09-03 11:27:59 ~38 min tests 📄log

@igor-sirotin
Copy link
Collaborator Author

The first group of tests is still timing out, even with 5 minutes per run 🤔

panic: test timed out after 1h40m0s

@igor-sirotin
Copy link
Collaborator Author

Alright, no timeout panics now:
image

@igor-sirotin igor-sirotin marked this pull request as ready for review September 2, 2024 23:18
@igor-sirotin igor-sirotin requested review from osmaczko and a team September 2, 2024 23:19

if [[ $HAS_PROTOCOL_PACKAGE == 'false' ]]; then
# This is the default single-line flow for testing all packages
# The `else` branch is temporary and will be removed once the `protocol` package runtime is optimized.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will stay with us for a while :D

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

true 🥲

_assets/scripts/test.sh Outdated Show resolved Hide resolved
@igor-sirotin igor-sirotin force-pushed the fix/protocol-test-separate-process branch from f209106 to e8d67cd Compare September 3, 2024 10:48
@igor-sirotin igor-sirotin merged commit 7623f68 into develop Sep 3, 2024
11 checks passed
@igor-sirotin igor-sirotin deleted the fix/protocol-test-separate-process branch September 3, 2024 11:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Nightly tests coverage is broken
3 participants