Welcome to video-simili-duplicate-cleaner Discussions!

[theophanemayaud]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about.
• Share ideas.
• Engage with other community members.
• Welcome others and are open-minded. Remember that this is a community we
  build together 💪.

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[ern2150]

Howdy - I'm sorting through a large video library that has a lot of duplicate segments across multiple files, sometimes even inside the same file. I have no intentions of deleting anything, so I'm perhaps far outside the scope of your core functionality :)

I bought the program for MacOS and am really liking how effortless it is to identify the most similar files inside a directory structure, as well as how it deals well with quality differences.

I think what I'm looking for is a way to report on the segments that are duplicates in detail (kind of how you allude to in #16) in a database structure. From here I could sort those segments into popularity, length, etc.

Does the program ever reset its cache database? I'd like to avoid rework as much as possible between uses, but it's very fast as it stands. I wonder if I could export that cache database to something simple as csv...

[theophanemayaud]

So it sounds like you're saying the app doesn't currently look at individual frames, and instead hashes on the data itself? That's certainly more efficient :)

No no, it does look at individual frames, specifically ! But depending on the mode, it might only look at 2 frames. Therefore it couldn't detect differences / similarities at other positions. For that, it'd need to capture more frames and compare more of them :)

Basically it's so I can easily export that data elsewhere, like to a google sheets doc where I can sort and chop and re-report on it and share it etc. If there's a more accessible cross-platform consumer tool for directly consuming the database, I'm all ears!

So what info would you want to be exported ?

[theophanemayaud]

Yeah in #16 you allude to another tool (video comparer pro) that seems to separate matches by segment (and report on percent match and duration of match), but maybe I was just seeing what I wanted to see rather than understanding what it really did.

Yeah I don't personally know this software, someone mentioned it to me. But anyways, I'm sure what you mean now !

[ern2150]

So what info would you want to be exported ?

Great question.
For each repeated segment:

• Some sort of "thumbprint id" that would be consistent across the files scanned so far*
• Duration of the segment
• A list of matches for this segment, and for each match:
  • Filename and starting timecode

*This seems easier than having it be a UUID as I initially thought, as long as I can "tag" it as a known segment I can associate the IDs in future scans by hand if necessary.

[theophanemayaud]

Hmmm, I'll keep this in mind but it seems very very specific edge case. I'll keep this here for now, until I decide to implement the "finding similar clips inside of videos" features !

[ern2150]

You know I’ve realized after doing some digging that ffmpeg does what I want, just that it takes quite a bit of massaging to make the data meaningful. If I end up generating a helpful script for that I’ll post a link here. Thank you for the discussion!

[ern2150]

The wilder idea that I'm building toward is to form a sort of infinite playlist across all these videos, using the common segments as pathways between them to keep the journey going.

For example:
Video 1 (segments of 5 or so seconds identified by letter):
ABCDEFZYXWVUT

Video 2:
ABGHIJKLMNZYC

Video 3:
ABOPQRSTUVZY

You could start at Video 3 at A, but jump to Video 2 at B, walk forward til Z and then jump to Video 1 at Y.
You could then wrap back to Video 3 at T, jump to Video 2 at Y, then back to Video 1 at C.

What gave me the idea is Spotify's Infinite Jukebox (which does the same thing but only inside one song).

[theophanemayaud]

That does seem a good idea ! For me it's not something I think would be easy 😅

[ern2150]

I think the other high-level question I wanted to ask is if you'd consider a command line interface for the tool as it stands? That might already be in an issue and I missed it :)

[theophanemayaud]

No, I haven't considered it for now. How would you want to work ?

[ern2150]

Replicating the main functions of the UI, just in a text format that can then be parsed. One example: you give the command a directory, a method, and a threshold, and it returns a paragraph of output per comparison. Instead of paging through that data in the UI, it’s generated as text, just missing the screenshots.

[theophanemayaud]

That is so the app could be used programmatically ? I guess it could make sense, yes, but I'll see if others ask for it because it might be a bit of work to make it export in a standard way...

[ruloes]

After many hours and searching with various programs, I continue to investigate what happens to those videos, whether they are corrupt or not and why the video comparer software gave errors.

I have detected looking through the folders that the following file gives an error, although as you can see in the capture I have checked it with HBBATCHBEAST and it is fine.

The days of testing with the program are passing and I cannot scan the disk completely because there are many files like this that I attach.

What can I do? Is there a program that detects these bad files and can exclude them from the analysis.

I know I am being very heavy but I need to scan the entire disk. These files were some recovered with recovery software type recuva, ... and I don't know which will be right and which will be wrong.

Thanks again for your understanding.

[ern2150]

Oh I understand, sorry you went through all that. If it’s ffmpeg under the covers that is failing awkwardly, then that’s not really something you would be expected to overcome.

[ruloes]

Thanks and can you help me??

…

________________________________ De: ern2150 ***@***.***> Enviado: martes, 14 de septiembre de 2021 10:58 Para: theophanemayaud/video-simili-duplicate-cleaner ***@***.***> Cc: ruloes ***@***.***>; Comment ***@***.***> Asunto: Re: [theophanemayaud/video-simili-duplicate-cleaner] Welcome to video-simili-duplicate-cleaner Discussions! (Discussion #14) Oh I understand, sorry you went through all that. If it’s ffmpeg under the covers that is failing awkwardly, then that’s not really something you would be expected to overcome. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#14 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AQMSU7Z5Q5SGYXTYNNQIQ7DUB6EMVANCNFSM42LFCDWA>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[ern2150]

Well I tend to think of things from a script/console perspective so I might not be the quickest help.
I was thinking you could do the mass ffprobe command on a directory, and then parse the output for problematic entries.
The trick is knowing which attribute to probe to be able to distinguish healthy resources from “sick” ones.

[ruloes]

I thought it would be something easier. At the moment I use a windows or macosx program but not scripts. How to distinguish those corrupted video files?

…

________________________________ De: ern2150 ***@***.***> Enviado: martes, 14 de septiembre de 2021 11:05 Para: theophanemayaud/video-simili-duplicate-cleaner ***@***.***> Cc: ruloes ***@***.***>; Comment ***@***.***> Asunto: Re: [theophanemayaud/video-simili-duplicate-cleaner] Welcome to video-simili-duplicate-cleaner Discussions! (Discussion #14) Well I tend to think of things from a script/console perspective so I might not be the quickest help. I was thinking you could do the mass ffprobe command on a directory, and then parse the output for problematic entries. The trick is knowing which attribute to probe to be able to distinguish healthy resources from “sick” ones. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#14 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AQMSU73537IMQW23EKM2CILUB6FH5ANCNFSM42LFCDWA>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[soulplanet]

Hello! I just purchased through app store. Am I correct in seeing it will only allow for scanning duplicate videos within the same folder? My reason for purchase is that I intend to combine volumes back together after I had to backup items very quickly and wherever I had the space, before a hard drive quit working. I know there is plenty of folder compare and merge software, but with video editing and potential similar videos at different resolutions etc.. I wanted something that was specifically video focused. So my questions ado I have to dump all files into a single folder on one volume in order to duplicate scan?

Secondly, above I saw you mentioned scanning more then a few video frames for better identification. Which setting do I adjust to do that?

Lastly, I don't claim to know anything about software design and app creating, but I know it's not as easy as it is for end-users to throw around suggestions! So thank you for your hard work. It seems your program has a solid structure, because it doesn't flinch when I throw a large folder at it. I have a photo specific duplicate scan program that has been excellent for a long time (photo sweeper). The way it deals with Apple Photos is it shows the duplicates, asks me to open Photos program at the end once I've marked items for deletion, and then it creates an Album within Photos with all the duplicates labeled "Trash" or something. Then, if I remember right, as long as I delete them in a certain way using that album (so that I delete actual files and not just the album) they're gone and I don't have to search anything individually. Like I said, I'm no expert, but thought I could at least share that info, in case it may help you (and ultimately me) develop a work around for duplicates within Apple Photos. Thanks in advance for your time!

[theophanemayaud]

videos within the same folder

You should be able to drag & drop multiple folders onto the main screen, and it will scan them all ! Or use the "select a folder" button multiple times. They will be added to the list of folders to scan. So no, you don't have to put them all in the same folder.

more then a few video frames for better identification

See https://theophanemayaud.github.io/video-simili-duplicate-cleaner/#main-window-settings : this is the dropdown in the main window, which can be 2 (one start one end) or more throughout the video :)

photo sweeper

Yes I also use it a lot, and think their implementation of Apple Photos is a great way. I'm working to also be able to do it like this. I have figured out pretty much how, now I need to get it working with the security systems in place by apple. I don't have much time at the moment so that's why it hasn't moved much. But I will definitely get it working like this :D Thanks for the idea, I agree, it's great !

[newadventure079]

The documentation states that best results are achieved with Thumbnails of 2x2. If that's the case, why does the program default to CutEnds?

[theophanemayaud]

Oh my bad that is not clear from the docs. I’ll amend it.
In fact, cutEnds is the best tradeoff for speed while also covering most cases.
2x2 and the others are much slower because of how many more frames need to be taken. And they don’t increase the detection accuracy much.