Skip to content

Add depth unprojection support to batch renderer#2129

Merged
0mdc merged 34 commits intomainfrom
batch-renderer-depth-unprojection
Jun 25, 2023
Merged

Add depth unprojection support to batch renderer#2129
0mdc merged 34 commits intomainfrom
batch-renderer-depth-unprojection

Conversation

@0mdc
Copy link
Copy Markdown
Contributor

@0mdc 0mdc commented Jun 13, 2023

Motivation and Context

This changeset adds support for unprojecting depth within the batch renderer.

In this context, unprojected depth means that each pixel of the depth image represents the distance in meters from the camera, rather than the raw depth buffer.

To accomplish this, DepthUnprojection is now shared by the classic and the batch renderer. To preserve batch renderer isolation, it was moved to the batch renderer along with the shaders.

Summary:

  • Batch renderer supports CPU depth unprojection.
  • Shaders are now split between gfx and gfx_batch.
  • gfx shaders can reference gfx_batch shaders, but not the other way around.
  • DepthUnprojection is moved to gfx_batch

Screenshot_20230613_111905

How Has This Been Tested

Tested locally and on CI.
The following tests are added:

  • Unprojection test in GfxBatchRendererTest
  • Image-based depth sensor test in BatchReplayRendererTest

Types of changes

  • Docs change / refactoring / dependency upgrade
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have completed my CLA (see CONTRIBUTING)
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@facebook-github-bot facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Jun 13, 2023
Comment thread src/tests/CMakeLists.txt Outdated
@0mdc 0mdc mentioned this pull request Jun 13, 2023
11 tasks
@0mdc 0mdc marked this pull request as ready for review June 14, 2023 01:45
Copy link
Copy Markdown
Contributor

@mosra mosra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

22 comments, sorry, I'll stop now 😅

Comment thread src/esp/gfx/CMakeLists.txt Outdated
Comment thread src/esp/gfx/RenderTarget.h
Comment thread src/esp/gfx_batch/CMakeLists.txt Outdated
Comment thread src/esp/gfx_batch/Renderer.cpp Outdated
/* Combined view and projection matrices. Updated from updateCamera() */
Cr::Containers::Array<ProjectionPadded> cameraMatrices;
/* Unprojection for cameras. Updated from updateCamera() */
Cr::Containers::Array<Mn::Vector2> cameraUnprojections;
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't this be put into the Scene struct among all other per-scene data? Feels unnecessary to allocate a dedicated array for these.

Copy link
Copy Markdown
Contributor Author

@0mdc 0mdc Jun 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm inclined to leave this as-is until we have multi-sensor support because it'll be more straightforward to refactor per-sensor data.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it'll be more straightforward to refactor

I don't know :) There most probably will be another array, yes, but that's about it, its size and contents will be significantly different. In the context of this PR it's unnecessary noise.

Comment thread src/tests/CMakeLists.txt Outdated
Comment thread src/tests/BatchReplayRendererTest.cpp Outdated
Comment thread src/esp/gfx_batch/Renderer.h Outdated
Comment thread src/esp/gfx_batch/Renderer.h Outdated
Comment thread src/esp/gfx_batch/Renderer.cpp Outdated
Comment thread CONTRIBUTING.md Outdated
Comment thread src/esp/gfx/CMakeLists.txt Outdated
Comment thread src/esp/gfx_batch/Renderer.h Outdated
Comment thread src/shaders/gfx/Shaders.conf Outdated
Comment thread src/tests/GfxBatchRendererTest.cpp Outdated
Comment thread src/esp/gfx_batch/DepthUnprojection.cpp
Copy link
Copy Markdown
Contributor

@mosra mosra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final set of code golf suggestions because this was bothering me :P

Comment thread src/tests/GfxBatchRendererTest.cpp Outdated
Comment thread src/tests/GfxBatchRendererTest.cpp Outdated
Comment thread src/tests/GfxBatchRendererTest.cpp Outdated
Comment thread src/tests/GfxBatchRendererTest.cpp Outdated
Comment thread src/tests/GfxBatchRendererTest.cpp Outdated
Comment thread src/esp/gfx_batch/DepthUnprojection.h
@mosra mosra force-pushed the batch-renderer-depth-unprojection branch from 1ae1473 to b100aad Compare June 25, 2023 10:51
@mosra
Copy link
Copy Markdown
Contributor

mosra commented Jun 25, 2023

Did the two remaining requested changes myself, see the commits below for details.

@mosra mosra force-pushed the batch-renderer-depth-unprojection branch 2 times, most recently from 58f56ed to ec15aac Compare June 25, 2023 11:32
mosra added 2 commits June 25, 2023 13:44
My suggestion was meant to *replace* the original and adapt the code as
necessary, not add a new overload without actually testing any of it.

Due to the copypaste the new overload also had no FMV attribute applied,
meaning it suddenly got significantly slower for no reason.
@mosra mosra force-pushed the batch-renderer-depth-unprojection branch from ec15aac to 46b8286 Compare June 25, 2023 11:44
@0mdc 0mdc merged commit 181aea8 into main Jun 25, 2023
@0mdc 0mdc deleted the batch-renderer-depth-unprojection branch June 25, 2023 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed Do not delete this pull request or issue due to inactivity.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants