src: Sort files in built beipack #20939
Conversation
`cockpit-bridge.beipack.xz` is currently not reproducible [1] as the order of files in it differs. Dictionaries are insertion-stable/reproducible since Python 3.7, so apparently the order of files in (`cockpit-0-py3-none-any.whl`) depends on the environment (we don't ship that wheel itself in distro packages). Sort the file names to make the content reproducible. [1] https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/cockpit.html
|
The F41/C10 regressions are unrelated and in the distros. Seen in fedora-selinux/selinux-policy#2318 or cockpit-project/bots#6775 . |
| @@ -86,7 +86,7 @@ def write(filename: str, data: AnyStr) -> None: | |||
|
|
|||
| def beipack_self(main: str, args: str = '') -> bytes: | |||
| from cockpit._vendor.bei import beipack | |||
| contents = {name: wheel.read(name) for name in wheel.namelist()} | |||
| contents = {name: wheel.read(name) for name in sorted(wheel.namelist())} | |||
There was a problem hiding this comment.
I think this is the wrong spot. I imagine the non-deterministic ordering comes from the os.walk() call in find_sources(). Then the files get added to the wheel in a non-deterministic order, and we iterate the wheel (getting the same order, I'd assume).
Two possible ideas for a fix:
- change
find_sources()to no longer be a generator but to build a list internally, calling.sort()on it before returning it; or - (and this is the better one, I think): add
sorted()around the two invocation sites offind_sources()where it matters: inbuild_wheel()andbuild_sdist(). Forfind_copy()it's irrelevant.
If you want to vary reading dirs you can toy around with |
|
Closing in favor of #20943, thanks @allisonkarlitskaya ! |
cockpit-bridge.beipack.xzis currently not reproducible [1] as the order of files in it differs. Dictionaries areinsertion-stable/reproducible since Python 3.7, so apparently the order of files in (
cockpit-0-py3-none-any.whl) depends on the environment (we don't ship that wheel itself in distro packages).Sort the file names to make the content reproducible.
[1] https://tests.reproducible-builds.org/debian/rb-pkg/unstable/amd64/diffoscope-results/cockpit.html
It's not obvious to me how to reproduce the different file layout -- I tried different build paths and different
LANG/LC_ALL/LANGUAGEvalues, which are the most obvious candidates from the build environment diff. In all cases I got the same beipack. However, the order (of course) does change with thissorted()patch.