Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(test_stats)_: better stats, print skipped tests #5781

Merged
merged 2 commits into from
Aug 29, 2024

Conversation

igor-sirotin
Copy link
Collaborator

@igor-sirotin igor-sirotin commented Aug 29, 2024

Description

Many thanks to GitHub Copilot 🙂

  1. Print skipped tests
  2. Print total count for failing and skipped tests
  3. For skipped tests, print the reason of skipping
  4. For failed tests, skip the root test, if a subtest is failing.
  5. 1 digit precision for failure rate

Looks like this now

---
Failing tests stats (total: 52)
---
TestSimulationBloomFilter: 99.0% (99 of 100 failed)
TestMakeNodeDefaultConfig: 53.0% (53 of 100 failed)
TestLoginWithKey: 50.0% (1 of 2 failed)
TestMessengerCommunitiesTokenPermissionsSuite/TestReevaluateMemberTokenMasterRoleInOpenCommunity_ERC721: 50.0% (1 of 2 failed)
TestMessengerContactRequestSuite/TestAliceOfflineRetractsAndAddsCorrectOrder: 50.0% (1 of 2 failed)
TestMessengerContactRequestSuite/TestAliceTriesToSpamBobWithContactRequests: 50.0% (1 of 2 failed)
TestMessengerContactRequestSuite/TestReceiveAcceptAndRetractContactRequest: 50.0% (1 of 2 failed)
TestMessengerContactRequestSuite/TestReceiveAndAcceptContactRequest: 50.0% (1 of 2 failed)
TestMessengerContactRequestSuite/TestAliceRecoverStateReceiveContactRequest: 50.0% (1 of 2 failed)
TestMessengerContactRequestSuite/TestAliceRestoresOutgoingContactRequestFromSyncInstallationContactV2: 50.0% (1 of 2 failed)
TestMessengerContactRequestSuite/TestBuildContact: 50.0% (1 of 2 failed)
TestMessengerContactRequestSuite/TestReceiveAcceptAndRetractContactRequestOutOfOrder: 50.0% (1 of 2 failed)
TestMessengerContactRequestSuite/TestAliceResendsContactRequestAfterRemovingBobFromContacts: 50.0% (1 of 2 failed)
TestMessengerContactRequestSuite/TestAcceptCRRemoveAndRepeat: 50.0% (1 of 2 failed)
TestMessengerContactRequestSuite/TestReceiveAndAcceptContactRequestTwice: 50.0% (1 of 2 failed)
TestMessengerContactRequestSuite/TestAliceRecoverStateSendContactRequest: 50.0% (1 of 2 failed)
TestMessengerContactRequestSuite/TestBlockedContactSyncing: 50.0% (1 of 2 failed)
TestMessengerContactRequestSuite/TestBobSendsContactRequestAfterDecliningOneFromAlice: 50.0% (1 of 2 failed)
TestMessengerContactRequestSuite/TestAliceOfflineRetractsAndAddsWrongOrder: 50.0% (1 of 2 failed)
TestMessengerContactRequestSuite/TestBobRestoresIncomingContactRequestFromSyncInstallationContactV2: 50.0% (1 of 2 failed)
TestMessengerContactRequestSuite/TestAcceptLatestContactRequestForContact: 50.0% (1 of 2 failed)
TestMessengerContactRequestSuite/TestPairedDevicesRemoveContact: 50.0% (1 of 2 failed)
TestMessengerContactRequestSuite/TestAliceSeesOnlyOneAcceptFromBob: 50.0% (1 of 2 failed)
TestMessengerContactRequestSuite/TestDismissLatestContactRequestForContact: 50.0% (1 of 2 failed)
TestPeerExchange: 11.1% (1 of 9 failed)
TestPeerPoolSimulationSuite/TestUpdateTopicLimits: 9.5% (2 of 21 failed)
TestPeerPoolSimulationSuite/TestMailServerPeersDiscovery: 9.1% (2 of 22 failed)
TestSyncDeviceSuite/TestPairingThreeDevices: 8.0% (2 of 25 failed)
TestPeerPoolSimulationSuite/TestPeerPoolCacheEthV5: 4.5% (1 of 22 failed)
TestSyncDeviceSuite/TestPairingSyncDeviceClientAsSender: 4.0% (1 of 25 failed)
TestSyncDeviceSuite/TestPairAcceptContactRequest: 4.0% (1 of 25 failed)
TestSyncDeviceSuite/TestPairPendingContactRequest: 4.0% (1 of 25 failed)
TestSyncDeviceSuite/TestPairingSyncDeviceClientAsReceiver: 4.0% (1 of 25 failed)
TestSyncDeviceSuite/TestPairDeclineContactRequest: 4.0% (1 of 25 failed)
TestSyncDeviceSuite/TestPreventLoggedInAccountLocalPairingClientAsReceiver: 4.0% (1 of 25 failed)
TestShhExtSuite/TestMultipleRequestMessagesWithoutForce: 4.0% (4 of 100 failed)
TestSubscriptionRemove: 3.2% (1 of 31 failed)
TestSubscriptionGetData: 3.1% (1 of 32 failed)
TestSubscriptionGetError: 3.1% (1 of 32 failed)
TestSQLLitePersistenceKeysStorageTestSuite/TestKeysStorageSqlLiteGetMissing: 2.0% (1 of 50 failed)
TestSQLLitePersistenceKeysStorageTestSuite/TestKeysStorageSqlLite_Count: 2.0% (1 of 50 failed)
TestPersistenceSuite/TestGetCommunityToValidateByID: 1.9% (1 of 53 failed)
TestPersistenceSuite/TestGetCommunityRequestsToJoinWithRevealedAddresses: 1.9% (1 of 53 failed)
TestPersistenceSuite/TestAllNonApprovedCommunitiesRequestsToJoin: 1.9% (1 of 53 failed)
TestPersistenceSuite/TestGetCommunityRequestToJoinWithRevealedAddresses: 1.9% (1 of 53 failed)
TestPersistenceSuite/TestCuratedCommunities: 1.9% (1 of 53 failed)
TestPersistenceSuite/TestDecryptedCommunityCache: 1.9% (1 of 53 failed)
TestPersistenceSuite/TestDeleteCommunitySettings: 1.9% (1 of 53 failed)
TestPersistenceSuite/TestGetCommunityRequestsToJoinRevealedAddresses: 1.9% (1 of 53 failed)
TestPersistenceSuite/TestDecryptedCommunityCacheClock: 1.9% (1 of 53 failed)
TestPersistenceSuite/TestGetCommunitiesSettings: 1.9% (1 of 53 failed)
TestShhExtSuite/TestRequestMessagesSuccess: 1.0% (1 of 100 failed)
---
Skipped tests (total: 30, skipped as flaky: 7)
---
TestOwnerWithoutCommunityKeyCommunityEventsSuite/TestOwnerControlNodeHandlesMultipleEventSenderRequestToJoinDecisions: flaky test
TestTokenMasterCommunityEventsSuite/TestTokenMasterControlNodeHandlesMultipleEventSenderRequestToJoinDecisions: flaky test
TestAdminCommunityEventsSuite/TestAdminControlNodeHandlesMultipleEventSenderRequestToJoinDecisions: flaky test
TestMessengerCommunitiesTokenPermissionsSuite/TestBecomeMemberPermissions: emitter is closed
TestMessengerCommunitiesTokenPermissionsSuite/TestRequestAccessWithENSTokenPermission: unknown reason
TestMessengerBackupSuite/TestBackupSettings: flaky test
TestMessengerPeersyncingSuite: unknown reason
TestMessengerStoreNodeCommunitySuite: requires storev3 node
TestMessengerStoreNodeRequestSuite: requires storev3 node
TestMessengerSyncSettings/TestSyncSettings_StickerPacks: Currently sticker pack syncing has been deactivated, testing to resume after sticker packs works correctly
TestSyncDeviceSuite/TestTransferringKeystoreFilesAfterStopUisngKeycard: flaky test
TestResolver: skip test using infura
TestGetName: skip test using infura
TestOwnerOf: skip test using infura
TestContentHash: skip test using infura
TestPublicKeyOf: skip test using infura
TestAddressOf: skip test using infura
TestExpireAt: skip test using infura
TestPrice: skip test using infura
TestResourceURL: skip test using infura
TestGetFilterLogs: Skipping due to flakiness: https://github.com/status-im/status-go/issues/1281
TestRaribleClientIntegrationSuite/TestAPIKeysAvailable: integration test
TestRaribleClientIntegrationSuite/TestSearchCollectibles: integration test
TestRaribleClientIntegrationSuite/TestSearchCollections: integration test
TestRequestPermission: skip test using infura
TestWeb3Call: skip test using infura
TestWeb3Signature: skip test using infura
TestPeerCount: flaky test
TestWakuV2Filter: flaky test
TestWakuV2Store: deprecated. Storenode must use nwaku

@igor-sirotin igor-sirotin changed the title chore(test_stats)_: better stats, list skipped tests chore(test_stats)_: better stats, print skipped tests Aug 29, 2024
@igor-sirotin igor-sirotin self-assigned this Aug 29, 2024
@status-im-auto
Copy link
Member

status-im-auto commented Aug 29, 2024

Jenkins Builds

Commit #️⃣ Finished (UTC) Duration Platform Result
✔️ 9b1b293 #1 2024-08-29 13:36:05 ~4 min ios 📦zip
✔️ 9b1b293 #1 2024-08-29 13:36:41 ~5 min tests-rpc 📄log
✔️ 9b1b293 #1 2024-08-29 13:36:59 ~5 min linux 📦zip
✔️ 9b1b293 #1 2024-08-29 13:37:11 ~6 min android 📦aar
✔️ 9b1b293 #1 2024-08-29 14:11:33 ~40 min tests 📄log

Copy link
Contributor

@osmaczko osmaczko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I trust the copilot

@igor-sirotin igor-sirotin merged commit 28e7a36 into develop Aug 29, 2024
11 checks passed
@igor-sirotin igor-sirotin deleted the chore/improve_test_stats branch August 29, 2024 20:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants