Add depth visualization #30
Conversation
|
this seems to depend on this: |
|
This is depend on the graphdeco-inria/diff-gaussian-rasterization#3 |
|
May I confirm if the current branch does not include the capability to render depth? |
|
The render part can be seen in #5 in diff-gaussian-rasterization |
Thanks for your reply. I have read your proposed code modification, which involves changing the alpha-blending process from ∑Tαc to ∑Tαd. However, I have noticed that there are instances where there are only a few points with a large covariance on a uniformly colored floor or wall. It is possible that using this method in such areas could result in significant errors. |
|
There are indeed many artifacts in the depth maps of 3D-GS. I tried to improve the geometry of 3D-GS by adding depth loss in #5 in diff-gaussian-rasterization, which might enhance the rendering effects of 3D-GS. However, experimental results show that improving the geometry does not substantially improve the rendering quality. I personally believe that the depth issues of 3D-GS are inevitable. |
|
@ingra14m Thank you very much for your implementation of depth forward and direction, but when I pull your code and then try to do depth supervision, a memory leak occurs at a random step of the training iteration: What can be determined is that the error has nothing to do with the location where the error is reported：
When using the original rasterizer implementation, this does not exist. so the problem is in the depth gradient calculation part of your implementation, I don't know what's going on, are you facing a similar problem again? |
Replenish： error： |
That problem was fixed in the latest version of diff-gaussian-rasterization. If you want to visualize the depth in the forward pass, you can refer to my another branch latest and reset to commit with the comment "Add depth forward pass" in the following picture. The latest commit of that branch containing visualization of the acc and depth. |
Thank you for your response! , I tried pulling the latest latest branch:
Training progress: 0%| | 0/30000 [00:00<?, ?it/s] Your implementation also seems to conflict with the official implementation of 3D-GS on the output of the rasterizer. Can you tell me how to solve it? |
You can |
|
when I was running depth supervision on your modified code, I also faced a similar problem about It seems like the problem in the code is the depths computation of |
Thanks for your reply, but I want to back-transmit the depth gradient for depth supervision, so can you tell me how you solved the memory leak problem to get the results after depth supervision? when I using latest branch and change rasteriser return: rendered_image, radii, depth, acc, Training begins，but the same error occurred： There is a memory leak in the calculation of DL_Ddepth but I can't solve it, can you give me some helpful suggestions? |
This problem is solved by official diff-gaussian-rasterization. The 'latest' branch is build on the latest official diff-gaussian-rasterization. The latest commit of the latest added acc visualization. So if you want to back-transmit the depth gradient for depth supervision, there are two choices:
rendered_image, radii, depth, acc = rasterizer(
means3D = means3D,
means2D = means2D,
shs = shs,
colors_precomp = colors_precomp,
opacities = opacity,
scales = scales,
rotations = rotations,
cov3D_precomp = cov3D_precomp) |
I think this memory problem is solved by official 3D GS. You can refer to my another branch, this branch build upon the latest diff-gaussian-rasterization. Since I add acc visualization, you can |
|
I used the second solution when the problem mentioned above occurred, and the memory leak problem was still visible in my experiments.
RuntimeError: CUDA error: an illegal memory access was encountered |
|
I recommend you to reset, because the recent commits related to acc have not been strictly checked. |
Frustrating that it still happens after a reset, what should I do? |
|
@ingra14m Sorry to bother you! |
|
sorry for late reply. Could you provide me with the dataset you used? |
|
@ingra14m can't provide the dataset, but exact same error happens to me as well. Will attaching snapshots help? |
|
It would be definitely great if you could provide a snapshots of the errors. |
Thank you very much for your reply! In my test, all data sets will have memory leakage errors. My depth comes from the monocular depth estimation algorithm. I have tried a lot of data sets, including mipinerf-360 scenarios. I will upload the dpeth of the data set to the cloud disk tomorrow. I am looking forward to your experimental results.! Thank you again! On the other hand, I got a snapshot of the error and turned it on. He checked the output and input, but no abnormality was found. |
Here is a very simple custom scene I made and used to test deep supervision： |
|
got it. I'll check it right away |
Thank you for your patience. I have completed the testing of your data. Fortunately, your data can be executed on my end, and the rendering results with depth are evidently much superior to those without depth. Here is my core code: depth = render_pkg_re["depth"]
# Just make the depth_gt and the gaussian depth with the same dimension
depth_image = depth_image.reshape(1, 1, *depth_image.shape)
depth_gt = F.interpolate(fid, size=(900, 1600), mode='bilinear', align_corners=False)[0] # match the scaling process of official 3D Gaussian
depth_gt = depth_gt / depth_gt.max() # make the depth_gt ranging from 0-1
loss = (1.0 - opt.lambda_dssim) * Ll1 + opt.lambda_dssim * (1.0 - ssim(image, gt_image)) + l1_loss(depth_gt, depth) * 0.1As for the Differential Gaussian Rasterization, I did as I said before (I have made a depth branch): git clone https://github.com/ingra14m/diff-gaussian-rasterization
cd diff-gaussian-rasterization
git checkout depth
pip install .Here is the results from me:
with depth supervision without depth supervision |
|
I didn't do anything special. I just read the corresponding depth in your depth directory. I think if there's anything I can call special, it's probably my depth normalization |
|
I quickly tried it, but as always, I couldn't perform the full training anyway, and the error kept appearing randomly in the middle of the training: RuntimeError: CUDA error: an illegal memory access was encountered I thought I might have missed some key step, but I did install the rasterizer as you asked: then: CUDA_VISIBLE_DEVICES=2 python train.py -s data/test_depth_super -r 4 I did some more tests, and I found that if the gap between gt and pred is very large when calculating loss, corresponding to the situation where gt is not processed, a memory leak will quickly occur, so I think the problem is gradient explosion. caused. In the case of inputting real gt, this problem does not exist, because after normalization, the difference between the two is very small. |
|
Cloning from the original 3DGS repo and doing: I get the same error: File "train.py", line 96, in training |
|
Hi @ingra14m, I also changed the branch to "depth" and reinstalled it through pip.
I am also using V100, would it be possible for you to directly zip the whole "gaussian-splatting" folder that can run this dataset for me to double-check? |
I think if you don't convert pred_depth to disparity map to calculate the loss, then the algorithm will diverge quickly and should not get good results but should be completely unoptimizable, so I'm confused about the results you got, you can share the checkpoints with me ? |
|
I must sincerely apologize, as I have to admit that I made a mistake. In your dataset, I used the wrong environment. The results I obtained were actually using the main branch of diff-gaussian-rasterization, a branch that only has a forward pass for depth and lacks a backward pass. Theoretically, adding depth loss in this environment should have no impact on the results, as there are no depth-related gradients affecting the parameters of 3D Gaussians through backpropagation, but the experimental results indeed show an impact. After using the depth branch, I encountered the same illegal memory access error on your dataset. This is a situation I have never encountered before. In my experiments today, the HyperNeRF dataset and my custom Lego dataset can run normally, but errors occur with the Tensor4D and your dataset. I speculate that it might be due to the lesser number of viewpoints; otherwise, this is for a reason unknown to me. You can find my modifications in my commit, which theoretically should not produce illegal access issues. |
Thank you very much for your reply, I think the error is not due to too few views.
The algorithm seems to no longer make errors in his environment, although it is completely impossible to obtain the correct gradient, which seems to confirm my above guess. |
|
Thank you for your interest in this work. Although there are some issues with it at present, rendering it inapplicable to all datasets with depth, I have some suggestions that might aid your subsequent endeavors.
I hope your research progresses smoothly, and should you require assistance, I will do my utmost to help. |
|
Hi, I encounterred a similar issue. After reading his implementation (thanks to that), I found somewhere to be interesting. At here, the code style is python-like even though it should be cuda. I think that depth assignment should be included in the if-statement, therefore I added a big parentheses to include both for-statement and that line. After recompiling, the error disappeared in my case. Hope my modification follows the original idea. |
Thank you very much for your discovery. It is evident that I employed Python styling in the code and overlooked the fact that this is CUDA code lol. I appreciate your identification of this issue; this bug has now been corrected. I wish you smooth progress in your research! |
|
@ingra14m can you point out some derivation of the depth gradient computation? |
@VladimirYugay As illustrated in Equation (3) of the 3D-GS paper, the color for each pixel is |
Hi, I am not sure if it is correct. In your implementation, the depth is defined as the weighted aggregation of z-value in the camera space. But, in other nerf implementations, I think the depth is usually defined as the weighted aggregation of the length between the points on the ray and the origin of the ray? |
|
@RaymondJiangkw Hi, |
Thanks for your explanation. :) |
Merge the latest
Thank you for the interesting idea, but I'm not sure it is really acceptable. |
Maybe you are right. This is the depth from drums in vanilla NeRF Dataset. But I'm not sure if the NeRF/Mip-NeRF can handle the depth of multiple transparent surfaces. |
|
@ingra14m Sorry to bother you. I try to render depth image using your given method. I use the followings: I am in main branch, because I only want to get the rendered depth. But when I try Thank you very much!!! |
A naive way to select the frontmost surface is to apply a low softmax temperature
|
|
@ingra14m |
hi, I think a possible reason is that the current URL of my repo is https://github.com/ingra14m/depth-diff-gaussian-rasterization. You may not have downloaded my repo completely. |
Hi @robot0321, because NeRF provides blender files, I directly used Blender's depth rendering engine to output the depth. I normalize this depth as GT. I've rendered the Lego and Hotdog with depth here if you need them. |
Could you please explain what is the difference between the distance of points on the ray to the camera origin and the z-coordinate in the camera space? Aren't they the same? |
If we simplify the situation down to 2d, with the ray shooting in the +x axis, the "distance of points on the ray to the camera origin" would be the hypotenuse, where as the "z-coordinate in the camera space" refers to the length of the x axis side. so to convert depth to a 3d point, assuming we just shoot the ray out and do not account for camera rotation and translation: I hope this makes sense. |
|
Hi，what a great job! |
Hi, the depth is used to determine the order in which the splats are added up. The SH coefficients are used to get the directional color when rendering in cuda (by default, unless you pass in a color override from python). Depth is for rendering order. Remember that the gaussians are first split into tiles, then in each tile they are added up by the C * alpha * T equation in order by increasing depth. This fork merely computes and returns the depth image with a very similar equation D * alpha * T. |
No description provided.