Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make e2e more robust #1128

Merged
merged 4 commits into from
Mar 4, 2025
Merged

Make e2e more robust #1128

merged 4 commits into from
Mar 4, 2025

Conversation

AlitaBernachot
Copy link
Collaborator

@AlitaBernachot AlitaBernachot commented Feb 20, 2025

Description

This PR introduces improvements to the e2e workflow, making the e2e tests less disruptive during development.

  • Turn beforeEach block into beforeAll (before in Cypress) => 🛑🛑🛑 KO
    Unable to switch to before (instead of beforeAll), Cypress does not store the state of the browser, it is mandatory to refresh/visit the page for each spec.
  • retry login to GeoNetwork => done when test are flaky, it will retry the login step and all what is declared in beforeEach for the current flaky spec
  • fine tune e2e launching with affected projects => updated Github workflow to launch e2e only if app is affected
  • flakyness => if a spec fails, it will be restarted 5 times max until it succeed.

As a result, here are the changes:

  • updated Cypress version v13.0.0 (not latest because of @nx/cypress)
  • added flaky tests handling (retries failing tests 5 times until one test passes), if a spec fails 5 times in a row, you can consider it as a real failure
  • updated Github workflow + fix on how to retrieve affected app (see env.AFFECTED_APPS)
  • ⚠️ cy.readFile() has been commented in datasetDetailPage.cy.ts => I can't find why this does not work with Cypress v13 anymore. It looks like Cypress freezes for this specific case. Intercept seems to not work when downloading on a tag with download attribute. Maybe one other way to test this without downloading the file, would be to check the value of the link and ignore the file content as this is the server responsability to produce the right file.

Architectural changes

The following library now depends on Cypress v13.0.0 (introducing experimentalStrategy for retries/flakyness).

Screenshots

Example of flaky tests
image

Now Github bot's comment is reflecting the right affected apps and libs
image

Quality Assurance Checklist

  • Commit history is devoid of any merge commits and readable to facilitate reviews
  • If new logic ⚙️ is introduced: unit tests were added
  • If new user stories 🤏 are introduced: E2E tests were added
  • If new UI components 🕹️ are introduced: corresponding stories in Storybook were created
  • If breaking changes 🪚 are introduced: add the breaking change label
  • If bugs 🐞 are fixed: add the backport <release branch> label
  • The documentation website 📚 has received the love it deserves

Copy link
Contributor

github-actions bot commented Feb 20, 2025

Affected libs:
database-dump,api-metadata-converter,api-repository,feature-catalog,feature-record,feature-router,feature-editor,feature-search,feature-map,feature-auth,ui-search,data-access-datafeeder,feature-notifications,util-data-fetcher,feature-dataviz,ui-dataviz,common-fixtures,ui-elements,ui-catalog,util-shared,ui-widgets,ui-inputs,ui-layout,ui-map,data-access-gn4,util-app-config,common-domain,util-i18n

Affected apps:
metadata-converter,metadata-editor,datahub,demo,webcomponents,map-viewer,search,datafeeder,data-platform

  • 🚀 Build and deploy storybook and demo on GitHub Pages
  • 📦 Build and push affected docker images

@AlitaBernachot AlitaBernachot force-pushed the e2e_more_robust branch 3 times, most recently from 5c243a6 to 5965a82 Compare February 20, 2025 15:30
Copy link
Contributor

github-actions bot commented Feb 20, 2025

📷 Screenshots are here!

@AlitaBernachot AlitaBernachot force-pushed the e2e_more_robust branch 25 times, most recently from 8025c1a to ed39bda Compare February 24, 2025 21:24
@AlitaBernachot AlitaBernachot force-pushed the e2e_more_robust branch 9 times, most recently from 1a1457d to d6924bc Compare February 25, 2025 18:14
@AlitaBernachot AlitaBernachot marked this pull request as ready for review February 25, 2025 18:15
@coveralls
Copy link

coveralls commented Feb 25, 2025

Coverage Status

coverage: 84.348% (-0.5%) from 84.847%
when pulling 34e3e64 on e2e_more_robust
into 22c177e on main.

@AlitaBernachot AlitaBernachot force-pushed the e2e_more_robust branch 8 times, most recently from cad7404 to 45dfc46 Compare February 26, 2025 17:24
Copy link
Collaborator

@cmoinier cmoinier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A big thanks for doing this work! It looks good to me. I'm sure it will improve our CI experience 😃

@Guillaume-d-o
Copy link
Collaborator

seems all good for me, i hope we will find a solution about the download link

@AlitaBernachot AlitaBernachot merged commit 80a8304 into main Mar 4, 2025
14 checks passed
@AlitaBernachot AlitaBernachot deleted the e2e_more_robust branch March 4, 2025 14:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants