Handle retired ZIM

[Popolechien]

L832R61EE failed with Incorrect values for zim_install: Science_in_the_bath_freshest-produce.en aimhi_geography.en

Looking at the configuration, I see that Science in the bath was selected twice, and indeed the configuration dashboard has it twice:

but also

Hovering on the later I can see two files that have almost the same name except that one is scienceinthebath_en_freshest-produce.en and the other Science_in_the_bath_freshest-produce.en (so just one uppercase difference).
So clearly the library has three files too many on offer.

I looked up download.kiwix.org and could only find one instance however: https://download.kiwix.org/zim/videos/scienceinthebath_en_freshest-produce_2022-01.zim

FWIW the recipe is here: https://farm.openzim.org/recipes/scienceinthebath_playlist

[rgaudin]

OK, there are two known issues here:

• catalog synchronization. The cardshop caches the catalog for 24h. I believe in this case, the catalog has changed recently and we shall see a single occurence of Science In The Bath tomorrow. This will be partly fixed by the move to OPDS (Use new tagging system #131) although we'd still use some cache. I suggest to shrink the current cache to 3h now.
• ZIM obsolescence. As far as the Cardshop is concerned, those are just two different ZIMs because they have different names so it's not a duplicate. The problem is (error is at hotspot which downloads the catalog on launch) that the referenced ZIM is not in the catalog anymore (meaning that its file is not downloadable anymore). We've already discussed this but I couldn't find the issue (or we didn't create one?). We need to define a practical way to handle such case:
  • silently remove the unavailable ZIM from the order and pursue (might be the only ZIM or content though!)
  • fail the order with an error message indicating a selected ZIM has retire and configuration must be updated.
  • something else?
    In addition, we need a system that checks for retired ZIM in configs and informs users about the issue. When? On config edit ? on Order placement ?

[kelson42]

@rgaudin Independently of any other consideration, the cardshop recipe should be checked as early as possible against the library opds stream... to avoid later crash.

Related to openzim/cms#7

[rgaudin]

Doesn't answer the question of the when except for the obvious “on order creation”. Just to be clear, the current check is at installer startup so first thing once the task is picked up by the worker. We'd want the cardshop to check that before scheduling the order… using OPDS, once we add OPDS support.

[kelson42]

@rgaudin Yes, Cardshop should be able to test a recipe periodicaly and even before a creator is started.

[rgaudin]