Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nightly failing as of Jan 30th #747

Closed
EveningStarlight opened this issue Feb 4, 2025 · 5 comments · Fixed by #748
Closed

Nightly failing as of Jan 30th #747

EveningStarlight opened this issue Feb 4, 2025 · 5 comments · Fixed by #748
Assignees

Comments

@EveningStarlight
Copy link
Contributor

Tests were passing on the 29th: https://github.com/StatCan/aaw-kubeflow-containers/actions/runs/13026284774
but failing on the 30th and thereafter: https://github.com/StatCan/aaw-kubeflow-containers/actions/runs/13047331140

The Test packages is flagging things that are not packages that we install, or have installed.
jupyter-resource-usage

@EveningStarlight
Copy link
Contributor Author

EveningStarlight commented Feb 4, 2025

Possibly related to the Ubuntu upgrade on the github runner
https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20250126.1

As the runner was successful on Version: 20250120.5.0 but not on Version: 20250126.1.0

@EveningStarlight
Copy link
Contributor Author

There's almost 50 more hits for the word python in the new failing test case

Image

Huge number of new packages being tested shown in the logs, see diff column

Image

And an example, showing a new test for pkg-config

Image

@EveningStarlight
Copy link
Contributor Author

EveningStarlight commented Feb 4, 2025

List of packages that are new as of the runner update:
Python (This case fails)
pkg-config
pyspark
jupyter-resource-usage

R (this case passes)
arrow
aws.s3
caTools
hdf5r
markdown
odbc
renv
sf
sparklyr

After removing all the diff from timestamps and container hashes, we are left with this diff, the tail end of additions all being the logging of the failure, more manageable than I was expecting

Image

@EveningStarlight
Copy link
Contributor Author

The weird thing is that some of the PRs are still passing builds on the new runner version,
see https://github.com/StatCan/aaw-kubeflow-containers/actions/runs/13120307190/job/36605760370?pr=746

@EveningStarlight
Copy link
Contributor Author

Okay, a new branch that reverted commits to base of staged builds still fails the tests.
#749

So the error will exist regardless of changes made to the build.
The fix still works, but nothing we added caused it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant