yum reposync consistently fails on a specific file

[verdverm]

We are trying to reposync this repository

yum-config-manager --add-repo https://copr.fedorainfracloud.org/coprs/g/oamg/leapp/repo/epel-7/group_oamg-leapp-epel-7.repo

but we consistently hit the following error during sync.

[MIRROR] leapp-repository-0.15.0-0.20220307164627737609.pr1.227.g3f3a37c.el7.src.rpm: Status code: 404 for https://download.copr.fedorainfracloud.org/results/@oamg/leapp/epel-7-x86_64/03635405-leapp-repository/leapp-repository-0.15.0-0.20220307164627737609.pr1.227.g3f3a37c.el7.src.rpm (IP: 18.172.134.128)
[2024-02-23T01:59:29.793Z] (17-19/179): leapp-r 18% [===-                ] 5.8 MB/s |  25 MB     00:19 ETA
(17/179): leapp-repository-0.19.0-0.20240213163  20 MB/s | 1.0 MB     00:00    
[2024-02-23T01:59:29.793Z] (18-19/179): leapp-r 18% [===-                ] 5.9 MB/s |  26 MB     00:18 ETA
[MIRROR] leapp-repository-0.15.0-0.20220307164627737609.pr1.227.g3f3a37c.el7.src.rpm: Status code: 404 for https://download.copr.fedorainfracloud.org/results/@oamg/leapp/epel-7-x86_64/03635405-leapp-repository/leapp-repository-0.15.0-0.20220307164627737609.pr1.227.g3f3a37c.el7.src.rpm (IP: 18.172.134.128)
[2024-02-23T01:59:29.793Z] (18-20/179): leapp-r 18% [===-                ] 5.9 MB/s |  26 MB     00:18 ETA
(18/179): leapp-repository-0.19.0-0.20240215090  24 MB/s | 1.0 MB     00:00    
[2024-02-23T01:59:29.793Z] (19-20/179): leapp-r 19% [===-                ] 6.1 MB/s |  27 MB     00:18 ETA
[MIRROR] leapp-repository-0.15.0-0.20220307164627737609.pr1.227.g3f3a37c.el7.src.rpm: Status code: 404 for https://download.copr.fedorainfracloud.org/results/@oamg/leapp/epel-7-x86_64/03635405-leapp-repository/leapp-repository-0.15.0-0.20220307164627737609.pr1.227.g3f3a37c.el7.src.rpm (IP: 18.172.134.128)
[2024-02-23T01:59:29.793Z] (19-21/179): leapp-r 19% [===-                ] 6.1 MB/s |  27 MB     00:18 ETA
[MIRROR] leapp-repository-0.15.0-0.20220307164627737609.pr1.227.g3f3a37c.el7.src.rpm: Status code: 404 for https://download.copr.fedorainfracloud.org/results/@oamg/leapp/epel-7-x86_64/03635405-leapp-repository/leapp-repository-0.15.0-0.20220307164627737609.pr1.227.g3f3a37c.el7.src.rpm (IP: 18.172.134.128)
[2024-02-23T01:59:29.793Z] (19-21/179): leapp-r 19% [===-                ] 6.1 MB/s |  27 MB     00:18 ETA
[FAILED] leapp-repository-0.15.0-0.20220307164627737609.pr1.227.g3f3a37c.el7.src.rpm: No more mirrors to try - All mirrors were already tried without success

[pirat89]

Hello @verdverm. It's possible as all builds are removed in ~2w automatically except the build with highest NEVRA. The removal of created (S)RPMs (to save a space on servers) most likely does not include update of YUM repositories. For more information about COPR, read: https://docs.pagure.org/copr.copr/index.html. In case you are considering this problematic, you can open bug/rfe for the COPR project to discuss it, but so far I expect that reposync of YUM repositories in COPR is not expected use case.

The particular repository is supposed to be used to install up-to-date upstream (devel) versions of leapp and leapp-repository packages + testing builds for opened PRs. From time to time (once per year(s)) I do a cleaning, dropping information about all obsoleted builds older than couple of months, which most likely removes also information about builds from repository metadata. However, I expect this does not resolve the problem with the syncing anyway. Why do you want to actually reposync this repository? If I can ask. Regarding the purpose of the repo, I cannot come up with a use case for that.

[verdverm]

The reason is that we have air-gapped systems. We sync and build tarballs for all repositories that we need where we cannot access the internet.

I would add that the removal of packages is another reason we internally mirror all repos. We try to pin versions, but if mirrors remove them, we cannot rely on those fixed versions being available. Thus we host our own mirrors built from reposync to ensure the packages we use remain available.

This is the only yum repo that I know of that removes packages (of the 16+ we mirror). Nvidia recently removed their centos7 repo all together, but older packages are still available in their centos8. So, as a user, I would generally expect that even outdated packages remain in a yum repo.

Regardless, I would expect a reposync should still work even with packages being removed. Does this sound like the index not being updated after deletion?

[pirat89]

Kind of "indexing" could be the problem, if you mean by it that repository metadata do not drop information about removed RPMs. From that point, you need to report it to the COPR project. I found there already some issues that may be related (https://github.com/fedora-copr/copr/issues?q=is%3Aissue+is%3Aopen+prune). Wouldn't be in your case than more reasonable to build packages by yourself when you need an older version? In both projects, we provide e.g. make build_container, so e.g.:

  BUILD_CONTAINER=el7 make build_container
  BUILD_CONTAINER=el8 make build_container

called in each project should create builds you want from the current commit. Well, from the purpose of the leapp copr repository, it's not expected to provide older builds available. Look at it kind of like rolling-distro. So unless you want to install a build for particular PR to test it, it's expected that people will use just the up-to-date upstream build. Older builds are persistently provided only in downstream (e.g. RHEL).

In case you plan to run that periodically, you can use also something like:

dnf reposync --repoid=<repoid> --newest-only

which will sync only the build with highest NEVRA, which means the latest upstream one. From what you wrote, I do not expect you have any use-case for devel builds created for particular PRs. So you are most likely interested only about builds with a release starting 100 (iow builds with substring master).

About the problem with non-existing RPMs in the repository (that are not removed from the repo metadata data), the only solution is to discuss the problem with the COPR project so the prune of COPR repositories clean also repo metadata. There is nothing we can do about that on our side and we do not plan to use anything else as COPR currently satisfies all use-cases we need covered.

Regarding that, in case nobody would merge or update any PR opened for leapp or leapp-repository projects for 2+ weeks, we expect you could find only 2 builds available in the repo - one for leapp and one for leapp-repository, so the reposync --newest-only should work always.

[verdverm]

There is nothing we can do about that on our side

You could not delete old packages like everyone else does not do. leapp-repository is the only package in the repository that does this deleting, in fact the leapp package has old versions available, so there is inconsistency from this project