Scanning get stuck

[stefan-123456]

Hello, I was used to run the program on a large (10 TB 10941 item) library. It was working fine but now stays stuck almost at the end . I tried to be patient and Waite 24 hours but no progress. Meanwhile it is eating CPU like crazy ....

How to find the file he gets stuck on ? The one showing in screen seems to be the last processed one but I cannot figure out in what order it it processing (alphabetically not, size not, date created not ...)

Any tricks I could use ? Thanks

[theophanemayaud]

Hi ! Thanks a lot for the feedback.
I’m so sorry for that ! It seems a bug has snuck in the last release but I still haven’t been able to pin it. I’ve had this same weird «stuck» situation happen also on my machines since the release, and it prevented me from releasing the next features I was working on, as I investigated. But I just was in the process of adding more logging options to more easily be able to track down such problematic files. I think I’ll give up on the bug hunt for now, release improvements to logging, and then we can diagnose better !
I’ll give updates this week, again so sorry.

[stefan-123456]

Hello Theophane, any progress ?

[theophanemayaud]

So sorry for not solving it already, I haven't had enough time yet in this Christmas period. I actually figured out that the bug I was seeing in my new features already exists and was just made more visible. So I started investigating this existing bug. But now I'm pretty sure that this is just the existing bug you also are seeing, so I will just publish my changes so we can better diagnose

[stefan-123456]

Hello, I still don’t see any new version … or may be can you put the previous version back, it was working at least for me.

…

Am 17.12.2024 um 14:34 schrieb Théophane Mayaud ***@***.***>: So sorry for not solving it already, I haven't had enough time yet in this Christmas period. I actually figured out that the bug I was seeing in my new features already exists and was just made more visible. So I started investigating this existing bug. But now I'm pretty sure that this is just the existing bug you also are seeing, so I will just publish my changes so we can better diagnose — Reply to this email directly, view it on GitHub <#137 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AS43IBSYZZUN6TC6EIQVS6L2GAR7VAVCNFSM6AAAAABTGDYQPSVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNJZGM2DOMA>. You are receiving this because you authored the thread.

[theophanemayaud]

Hey Stefan, so sorry again ! I just sent a new version 1.11.1 to apple a few hours ago, and it's not waiting for approval by apple. I'll let you know as soon as it is published, it should be a day or two (but maybe apple takes more time to review during Christmas time... ?) !

[bobergmans]

@theophanemayaud I noticed the update was available on the App Store, but the issue remains. Before it seemed to get stuck at 80-90% and now 13.223 of 13.237 files have been processed, but it still gets stuck. So unfortunately the update didn't fix the issue..

[theophanemayaud]

Ok thanks for the report ! So sorry. Could you enable «verbose» and see when it gets stuck, what are the STARTING files that haven’t yet had a SUCCESS or ERROR ? It’s processing multiple files at once so it will probably show many still «pendling». If you can isolate them and see if by scanning a folder just with those also gets stuck ? That would help a lot ! If it’s too many to sort through could you send me the log file at video-simili-duplicate-cleaner@outlook.com ? It contains all video names so that’s only if you are ok sharing that info with me. You can access logs folder via the settings menu.

I also just thought of a way to maybe circumvent the problem by implementing a timeout for each file processing, and will work on it tonight or tomorrow morning. It seems your problem is that for some videos it gets stuck and as soon as all parallel processes are stuck on such videos, nothing progressed further. So adding a timeout to clear up stuck processes could be a solution, and I could log what video it timed out with.

[stefan-123456]

Brilliant. Thank you theophane.

[stefan-123456]

Honnestly this is little help ... I ran it in Verbose mode and every time it stopped (Freezes) I removed from the folder the file shown in "Started" ... I did this 3 times, and it keeps stopping randomly. It is totally useless. Can't you bring back the version of September ?

[theophanemayaud]

I'm so sorry, the problem with the September version is that it was crashing for others :/
I've just submitted an update which should at least help us diagnose by logging which videos are "stuck processing", and at best circumvent the problem by aborting processing for those videos. It should be out in a few days when apple approves it.

Also, as you already identified some that were causing the stuck state, would you be able to send me one or two of those videos so that I can actually fix the problem ? I have tested on thousands of my test videos (long 2h+ movies and also all kinds of weird states) but have in the end not been able to reproduce the problem. Some strange videos take 10-20s to process, slowing down considerably the whole thing, but it never completely gets stuck. It's strange as I thought I also had your problem some time back, but now I'm not able to reproduce. It would help tremendously actually finding the root cause. If you can put them on a drive or something and send me the link at video-simili-duplicate-cleaner@outlook.com that would be amazing.

[stefan-123456]

Hello Theophane, sorry I cannot share the footages for copyrights reasons. I pulled back from a backup Video simili duplicate cleaner v1.10.1 and it works like a charm scans the whole drive in 10mn (12000 files) when the latest one runs 5 hours an then gets stuck ... So you definitely introduced the problem in the September version, you should have a look at where you modified your code because there is the issue.
The files where the latest version got stuck were not even big around 300mb and had nothing special (no smileys in file name or the like :-) .. ) Worst it would some time get stuck at file 5000 another time almost at the end 10500 another one at 7000 ... totally random :-(

[theophanemayaud]

Oh great that you had that backup !!! Yes of course no problem it was just in case those were videos you were able to share.
The thing is I really spent hours looking at the changes, and I didn't change anything relating to video processing... The only thing that changed was the version of the ffmpeg library, but it should only have been a bug fix change from them specifically addressing the crashes that other users were having for weird broken videos. I'll keep this issue in mind during the next goal, which is to move to even more updated ffmpeg versions which also hopefully fixes this.