Zimit2: Youtube videos are not playing (found at least in mesquartierschinois and 100rabbits)

[benoit74]

Page: https://dev.library.kiwix.org/viewer#mes-quartiers-chinois_fr_all_2024-01/mesquartierschinois.wordpress.com/2014/07/17/girona-menamora/

When you try to play the youtube video on this page, the video never starts.

[mgautierfr]

This is strange. The video exists in the zim file (youtube.fuzzy.replayweb.page/videoplayback%3Fid%3Do-AMT1WhCGLQGL0y0nkhuLvbKBBeOB1DzqsIIGuy4k6cYC) and is loaded by the page/js.
But we have a (recurrent) request to /www.youtube.com/api/stats/qoe%3F[lots_of_encoded_data] which is 404.

I'm investigating.

[benoit74]

Nota: I've updated the recipe exclude setting to (\/\?|\/\#|\/wp-admin) so that we scrape less way dummy things.

[Jaifroid]

I corroborate this -- I've seen it often while testing support for zimit2 in Kiwix JS and Kiwix PWA (that video is usually not working). A small example is in the 100 rabbits zimit2 file from 2024-01, article "Orca". I'm seeing a bunch of POST failures in console log on Kiwix PWA (see bottom right of screenshot below). I don't know if that's related.

There is a December version of the same archive using zimit classic, where the video plays fine (in the PWA or KJS Browser Extension). These two small ZIMs ("Orca" article) might make a useful test case for comparison.

The main difference is that when running in zimit classic, there are console messages saying "Attempting to resolve fuzzy headers for url...", whereas I don't see any such message in the zimit2 implementation.

In my old implementation where I did the fuzzy search in the reader, it tells me that it transformed the requested URL (in the zimit classic version) into this monster in order to get the MP4:

TRANSFORMED VIDEO URL C/A/youtube.com/embed/hQXa6TkSeH0 --> videoembed/C/A/rr1---sn-4g5lznl6.googlevideo.com/videoplayback?expire=1702613002&ei=qnt7ZdzMNZigi9oP2Peu6AE&ip=147.229.8.218&id=o-ABAAsNMIStDNtiuVafv009RPQXjyCOTdv1b5hIuNg3Ik&itag=18&source=youtube&requiressl=yes&xpc=EgVo2aDSNQ%3D%3D&spc=UWF9f8WFoK6W-cOhzhDsb0Eq-HaQsKY&vprv=1&svpuc=1&mime=video%2Fmp4&ns=Fo3yAF1efDhMZX7XGHmUKaMQ&cnr=14&ratebypass=yes&dur=75.302&lmt=1636994742695961&fexp=24007246,24350017,24350137,24350147&beids=24350017&c=WEB_EMBEDDED_PLAYER&n=avF1oDg1aQ5wDQ&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cxpc%2Cspc%2Cvprv%2Csvpuc%2Cmime%2Cns%2Ccnr%2Cratebypass%2Cdur%2Clmt&sig=AJfQdSswRgIhAJ1tE6EX30uDKCBWvDvJSrfwcXDLXbKVYRV81VsRAfnWAiEAnZvabbfIgoC6PI_IIptOCyGQYNQRDh0re9fIMxNa2TU%3D&cpn=AwzaoHrmYdSTbZgQ&cver=1.20231212.01.00&ptk=youtube_none&pltype=contentugc&cm2rm=sn-cgovpm-cg9l7l,sn-2gbek7s&req_id=1d838bb288c2a3ee&redirect_counter=2&cms_redirect=yes&cmsv=e&mh=kw&mm=34&mn=sn-4g5lznl6&ms=ltu&mt=1702591021&mv=m&mvi=1&pl=17&lsparams=mh,mm,mn,ms,mv,mvi,pl&lsig=AAO5W4owRgIhAKBI_a62ADXn1T4Nf1eczDBlt5WK_nd4ZsyNpGuy8qKcAiEA76eASYUWvqzjdnWxV8KkiYU9xCQfuJFNwJ4ucMW_A0A%3D

[Jaifroid]

Since the 100rabbits file (https://master.download.kiwix.org/zim/.hidden/dev/100rabbits_en_2024-01.zim) "orca" article video displays the exact same symptoms as mesquartierschinois, and I'm pretty sure I've seen it elsewhere too, maybe the title of this issue should be changed to "zimit2: youtube-based videos are unplayable in some instances". I'm not sure how widespread it is.

[benoit74]

Thank you @Jaifroid for finding the "Orca" issue.
Not working: https://dev.library.kiwix.org/viewer#100rabbits_en_2024-01/100r.co/site/orca.html
Working: https://library.kiwix.org/content/100r-off-the-grid_en_2023-12/A/100r.co/site/orca.html

[mgautierfr]

It appears we should apply the same transformation as https://github.com/webrecorder/wabac.js/blob/main/src/rewrite/dsruleset.js
As I have tested with warc generated by chrome extension and everything works with them without the domain specific transformation (I don't know why), I've missed it.

I will make a PR to add it.

[kelson42]

Fixed by #158