foreground off option or fix for any s model

[kaelsonofkrypto]

Hi some of my friends we have been testing more of the models and i have a specific test clip that i use to test everything with. its a difficult test clip that is less than 10 seconds. This test clip is good for seeing if artifacts happen around the bottom pews and also determining how edge fix and foreground helps or doesnt help. So on any large and zoe this specific test clip looks fine but on any s model even with selecting 0 and negative foreground we noticed that negative or zero foreground didnt work properly for the Any S model and was flat on the background. I love Any S model because its super duper fast and is faster than the other models, so please dont change the speed of any s, however i wanted to know if its possible to add a foreground off option for any s or fix it to where the foreground works like it works on the other models. Of course if this is going to affect the speed then dont change.

Here is the infamous 2d test clip u can use to test with to see what we mean https://drive.google.com/file/d/1e_USSt9YJaUtdwydYEsGXUvT_QqC-lEL/view?usp=drive_link

[nagadomi]

Any_S, Any_B, Any_L have different tradeoffs between speed and quality.

Model	Speed	Quality
Any_S	Fast	Low
Any_B	Middle	Middle
Any_L	Slow	High

The only difference is the number of parameters in the model, there is no special processing on iw3 side.
If you are not satisfied quality of Any_S, you should use Any_B or Any_L.

[nagadomi]

I cannot find any big difference between Any_B and Any_S in the above video.
Both outputs look almost the same.

foreground off

foreground-scale 0 for DepthAnything does nothing and uses the output of the model as-is.
In other words, foreground scale feature is turned off.
So I am not sure what the foreground off option means.

[kaelsonofkrypto]

hi , how can i revert back to a previous version of iw3? this is a good update on row flow, but for some reason i cant get the same speeds i was getting on 8k resolution, so i wanted to switch back so i can get back the speeds i was doing on 8k

[nagadomi]

row_flow_v3 has a different architecture from row_flow_v2. On my Linux env row_flow_v3 and row_flow_v2 are almost the same speed, but it may be slower on Window.
Is row_flow_v2 slower than before?
If you mean row_flow_v3 is slow, then use row_flow_v2, which is the same as the previous version.

---

To change the version, use git checkout <hash> command.

1. Run nunif-prompt.bat
2. Find the version and hash

git --no-pager log | more

This command shows the version history.
Press the space key to move to the next page and the q key to exit.
For example, the version before the recent stereo generation method change is this

commit d4b238502beca0c3796bbce9e7751f6bb761ebdd
Merge: f69ca66 07d6a8d
Date:   Sun Apr 21 15:46:16 2024 +0900

Then,

git checkout d4b238502beca0c3796bbce9e7751f6bb761ebdd

The version should have changed with this git checkout command.
To revert to the latest version,

git checkout master

Note that unless you run this command, update.bat will not be updated to the latest version.

[kaelsonofkrypto]

UPDATE; I just saw you did a new update 4 hours ago. I'm gonna try the new update you did and also see how that turns out. Thanks.

[kaelsonofkrypto]

Actually, you know what? I just checked again and no wonder you don't see the difference It is the same. Apparently, after the update, it changed because. I did a clip before the update and the people with cameras on the right side in the back used to be in 3D. but now they're not. Even Zoe is showing them as flat now and with Zoe, Zoe used to make everything 3D including the background as long as foreground was zero. The way it is acting on all the models is like foreground is on. The background used to be in 3D but now the background is flat.

Luckily, I actually have the clip. I did before the update I'm going to upload it to Google Drive so you can see how the same clip used to convert before the update. and now how it looks after the recent update you will notice the people on the right and the back with the cameras look flat after the update.

[nagadomi]

Maybe Depth Resolution is different than before?
I tried Any_L, Depth Resolution=518 or 770, Foreground Scale=-2 and I can see their depth.
With Depth Resolution=Default(392), it is difficult to see.

[kaelsonofkrypto]

Here is the. area to look for.

[kaelsonofkrypto]

This is before the update how the clip looked.

BEFORE recent update https://drive.google.com/file/d/1UOLeNV19oFE3s-tODdixD9AH5DWLXham/view?usp=drive_link

And this is after the recent update with same exact settings how it looked.

AFTER recent update
https://drive.google.com/file/d/19DguxW6jvFfRUrFlCLeeAJNs4ycxTVi5/view?usp=drive_link

Something changed in the most recent update with how it handles 3D in the background. It looks flat on the background after the most recent update. These two clips are the same exact settings. The only difference is one of the clips I did before the update and the other one I did after the update.. At first I thought it was just any S, but after checking other models. its the same exact thing. Foreground isn't behaving correctly like how it behaved in the previous version of IW3

[loawizard]

I just looked at the clips and i can see a slight difference in 3D. when we expirimented with the clip yesterday I saw on any large s that in my conversion the whole back row did not have 3D on any S. To bad because any S is very fast and almost no difference between default and 512 map. I need to do more conversions to see if it happends in different scenes or videos

[nagadomi]

This issue seems to have some confusion.
If you have noticed any problems using the latest version, let me know the exact settings.

Also, this project uses git, so we can easily test the version at a certain point in time.
If you tell me the specific date time, I can try that version or give you the command to switch to it.

[kaelsonofkrypto]

The settings i used was zoe any n, depth 0.85 , 512 depth resolution, 3 on your own size, converge 0.50, foreground 0, edge fix 1, TTA checked, fp16 unchecked, aspect ratio checked, zero latency unchecked, fast decode unchecked, batch size 1, threads 16.

I don't remember the specific time but what I can tell you was it was the version before you added negative foreground where it was more 3D ish on background people

[nagadomi]

The result with the settings provided is similar to BEFORE, not AFTER.
And AFTER looks like Flicker Reduction 0.99 or higher.
I think the differences in these videos are due to differences in settings, not software versions.

The cause of the flat problem is Flicker Reduction.
Flicker Reduction 0.99 can cause depth clipping for a few seconds after a scene change.
After the scene change, the audience is in 2D (flat) and takes 2-3 seconds to transition to suitable 3D.
Maybe it should detect scene changes and reset the state.

[kaelsonofkrypto]

i just did another test and changed the flicker to .75 and even .5 to see what your saying with flicker reduction and NOPE thats still not the cause of less depth on the background, the people on the right still dont look as 3d even on lowest flicker reduction settings in the newest update, maybe u can test the settings before u added adjustable flicker reduction but im very sure thats not the cause. because this issue didnt happen until after the update when the foreground negative options were added. before that option was added was when i did that before clip. im also certain because i tested that same clip over a 100 times with different depths different foregrounds and edge fixes and convergences to see what foreground does with different settings before the update occurred with the negative foreground options being added, previously before u added negative foreground, when u enable foreground it would make the background flat fixing warps in the background but it did so at the cost of making the background flat, when u disable foreground the background is 3d too. after the recent update Zero on foreground does not make the background 3d like it did before this last update where the negative foreground options were introduced

[nagadomi]

Also, am I right in assuming that it is ZoeD_Any_N that you are currently having problems with, and that Any_L and Any_S are unrelated?

[kaelsonofkrypto]

Thanks for your patience and responding.

From what i can tell, the issue is zero foreground is not behaving properly. Before the update a zero on foreground would allow the background to be 3d, if u wanted background to be flat then u select positive value of foreground . After the new update zero doesnt let the background remain 3d anymore, zero is functioning like positive foreground in previous version making the background flat. The example i provided had everyone in the seats 3d but the row behind them where the guards and the news reporters with the cameras they were flat so thats what i mean by background is the further object in the video being flat..

This issue is occurring on all models now, even zoe doesnt do the 3d on background properly like it did before. I never turned off flicker reduction on those before and after clips i showed. i used flicker reduction on both before and after. The before option was with it checked and the after was with it checked and upped to 99, but i have also done the tests u mentioned to see how it looks off and with .5 .75 after the update and it does not give the same 3d still like it used to be before update. So its not flicker reduction , this issue didnt happen until after negative foreground was added. i have foreground on zero but the clip is behaving like positive foreground values before the update. This is happening on zoe, any s, any b, and any large from what i tested

To be fair i did ask for negative foreground to see if it would help lol but it doesnt it causes wobble artifacts and it makes background flat, which that shouldn't have happened, it should be the opposite. it looked better before those options were introduced. I kinda wish i didnt ask for the negative foreground options now because zero on foreground is no longer behaving the same as it did before update 😂

Something else i notice, edge fix will fix some artifacts where the nose clips/cuts but then it introduces a halo artifact around the head if u do above 1. So 1 seems to be best for edge fix but at higher resolutions u can get away with doing zero on edge fix if u check tta.

[nagadomi]

I have already confirmed that Flicker Reduction is the cause of the problem of flat backgrounds.
If the problem you describe occurs with Flicker Reduction turned off, please let me know.

[kaelsonofkrypto]

Yes it still occurs with it off, i tried that too, thats what i was saying. It doesnt matter if u have it on or off, the people on that clip that are standing up behind the people who are sitting in chairs are flat still with flicker off or on compared to my version that was done before the update. The people in the chairs are fine, they are clearly in 3d , its the people behind them that are standing that are not in 3d. i tried off , .5 , .75, and .99 and no matter the setting on flicker those people standing in the back are flat. They never used to be flat like that in previous version

[nagadomi]

Ok, I'll check to see if there is any difference in the results between the 2024/4/1 version and the latest version.

[nagadomi]

I checked the difference between the 2024-04-01 version and the latest version,
and they are exactly the same with Flicker Reduciton: Off,Foreground-Scale: 0, Depth Model: {Any_L, ZoeD_Any_N}.

So if Foreground-Scale: 0 makes a difference from previous versions, it is related to Flicker Reduction.
Since you always had Flicker Reduction enabled, the output video is always affected by Flicker Reduction.

One of the reasons for the difference from previous versions I can think of is that I fixed the Flicker Reduction bug on 2024-04-04 8e7bb56.
This change only affects the first few seconds of the video, but if you were testing with a very short video it could have been affected.
Whatever it was a good affect, it is just a random result, so I won't revert this fix.

[kaelsonofkrypto]

I don't think it's a random result because I also have a video of the TV show sandman that I did with foreground zero and flicker reduction on and on Zoe model. It would always be 3D in the background but now same thing. Even on that video it's flat in the background now after the latest update , But I understand if you don't want to revert, but there's definitely a change I can see in the 3D with how the background is handled

[nagadomi]

If the video starts with 5-second previous scene, the change has no effect and the result should have been the same as the result of the current version.
The bug I fixed was that dummy frames unrelated to the video were being used in Flicker Reduction computation. The result is completely unpredictable.
So there is no point in sticking with this change.

[nagadomi]

Regarding the behavior of Flicker Reduction, I am planning to detect scene changes and reset the state, so it may change again in the future.

[kaelsonofkrypto]

Thanks so much for your help that's good to know that that's on the horizon, scene change definitely should help the issue

[kaelsonofkrypto]

hey hope your day is going well, got a quick question for you, does selecting low vram effect quality? reason i ask is because i notice if i select low vram it allows me to use higher depth resolutions that i couldnt normally do if i didnt check it.

[nagadomi]

Low VRAM option effects only VRAM usage and processing time (low vram usage and slow), not the quality of the results.
If the results differ, it is a bug.

[kaelsonofkrypto]

Thanks for confirming.

[kaelsonofkrypto]

Hi i just tested the new row flows, and row flow 3 sym actually performed the worst on my test clip, the regular row 3 performed the best, use this test clip and u will see what i mean https://drive.google.com/file/d/1e_USSt9YJaUtdwydYEsGXUvT_QqC-lEL/view?usp=drive_link

i think regular row flow 3 would be better as the default

[kaelsonofkrypto]

well i just learned something forward fill renders super duper slow on 8k on any b and any s its .02 fps so its unusable for 8K resolution on an rtx 3070, even on default its unusable for 8k resolution, 4K is the max that forward fill can be used on that is acceptable speeds

with row flow 3 i can do 8k at pretty fast speeds

[kaelsonofkrypto]

i just converted my 8k clip to 4k, so im testing the 4k clip with forward fill to see if it does better on this problem clip and will get back to u once i confirm which one is better between forward fill and regular v3 row flow

[nagadomi]

foward_fill uses very low quality inpainting, so it is not high quality (pixelization occurs). However, in theory, it behaves more correctly with respect to depth.

[nagadomi]

Its definitely slow, but it improved a bit yesterday. If you haven't updated it, update it.

[kaelsonofkrypto]

well i was wrong after more testing row flow 3 Sym is more accurate on faces on 512 depth resolution than forward fill and regular v3. however forward fill does better on some other types of objects, so there is no clear cut winner on depth resolution 512, i got a really big problem clip that shows a situation where sym was better than both forward fill and regular v3, and i have a clip where regular v3 and forward fill was better than sym lol.. i tested any b on 512 depth resolution with a 4k clip on row flow 2, forward fill, regular v3, and v3 sym and sym v3 was definitely more accurate on the face big time. the setting i used on all four clips was 0 on convg, 0 on edge fix, 0 on foreground, .99 flickerfree, depth 1.32 , depth resolution 512 and any b model on all four. in all my tests i use TTA FP32 (fp16 is unchecked)

I think however the winner will end up being regular v3 or sym v3 for one specific reason, regular v3 does have artifacts but its warping is less severe in some situations compared to the sym version and u will still be able to go to higher depth resolutions,, the higher depth resolutions will fix many of the artifacts that occur at 512 depth resolution. i didnt get any of these bad artifacts when i was doing depth resolution of 968 and 1020. so i feel these issues are occuring because i did the tests with 512 depth resolution. forward fill its impossible for me to go that high of a depth resolution, its too slow to do it at such a high resoution.

I however need to do one last test to really confirm, im going to test these same exact clips on any large model this time and report back later

[kaelsonofkrypto]

so after testing extensively, any large even didnt fix some artifacts at 512 depth resolution with this very difficult clip i have. the forward fill no matter how hard i tried did not fix a particular artifact on the face of this actor, regular v3 also had the same issue on the face. The sym version 3 was the only row flow model that actually fixed the face of the actor on my sandman clip however on another clip , my infamous argylle clip at the book signing, the pews in that scene had artifacts the sym model couldnt fix but was fixed by row flow regular v3.

To be fair v3 row flow is very similar to row flow 2 and in most situations it will mostly work on most artifacts. none of the row flows could fix the argyle clip but regular row flow v3 came the closest to fixing it. one thing i did see was with a high enough depth resolution, sym came in a close second to fixing the argyle pew scene. Because artifacts on a face are more distracting i guess i would have to give the edge to the SYM model. so i was indeed wrong, sym is the best model if u dont want artifacts on the face and body. but some areas like pews is where it may cause warping. In my experience, edge fix actually causes more artifacts, with edge fix i saw increased warping and halos so i dont use edge fix. forward fill is very similar to row flow v3 regular, the only difference is its pixels, and because of this i would give regular row flow 3 the edge but sym 3 indeed should remain the default after having performed a more thorough test.

this test clip only would convert properly only with SYM V3 on zero or negative foreground and zero edge fix, zero converge with flicker reduction at .99. I used depth 1.32 , here is the link to the test clip https://drive.google.com/file/d/1d7PBcOqw8eyVfWQY50u5Hdj2d73ECv9C/view?usp=drive_link every other row flow model including forward fill failed horribly on this test clip causing a buldge on the actors left cheek. test it for yourself to see.

[nagadomi]

I added row_flow_v3_rev2.
This has less left/right eye inconsistency(ghost artifact) than row_flow_v3.
The code to be executed is exactly the same as row_flow_v3. Only the training data differs (It uses forward_fill to generate synthetic training data).
So the result is very similar to forward_fill, but without the pixelization problem and with the same processing speed as row_flow_v3.

I plan to replace this with row_flow_v3 and make it the default method.

[loawizard]

i really like row flow v2 gives me best results with any B so far

[nagadomi]

row_flow_v2 has low pixel-level distortion, but cannot handle large divergence (3D Strength).
The main aim of v3 is to support 2.5-5.0 3D Strength. With higher 3D Strength, the best distance for the screen position is closer.
It is a very needed ability in some use cases, I think.

[kaelsonofkrypto]

Bro, thank you so much for the new update. the new updated rev 3 ver2 fixed my argylle test clip and negative foreground even looked proper on my argylle clip and made the people in the background 3d again. So i was happy i was able to finally fix that clip. However, it looks like this sandman clip is still struggling even on the new rev2. It appears on this particular clip. I could only get sym model of row flow to work. Would you mind testing this short 30 second clip nagadomi to see if you could figure out how to. get 2.0. depth looking clean on this test clip with rev 3 ver2. for what ever reason it causes a bulge in the face when i try to convert it to 3d. I think this would definitely be a good problem clip you could train on. to try to solve issues with it. https://drive.google.com/file/d/1d7PBcOqw8eyVfWQY50u5Hdj2d73ECv9C/view?usp=drive_link