ToF node enhancements, host-runnable ToF and image filters #1308

lnotspotl · 2025-04-28T06:28:00Z

The goal of this PR is to introduce a new host-runnable node for general image filtering (ImageFilters) as well as a new node called ToFDepthConfidenceFilter designed specifically for depth frame filtering based on data coming from the ToF node.

Additionally, the ToF node was renamed to ToFBase and a new ToF node took its place, combining ToFBase, ToFDepthConfidenceFilter and ImageFilters into a single node. This gives power-users the opportunity to fiddle with the low-level stuff (they can still use the origin ToF node implementation by instantiating the ToFBase class). For an average user, however, the new ToF node with a predefined set of filters will likely suffice.

The image filters were taken from the FW codebase and the ToF filter was implemented based on a reference python implementation provided by the CV team. Both the ToFDepthConfidenceFilter node and the ImageFilters node allow for run-time parameter updates thanks to the possibility of sending in input configs. In addition to that, the ImageFilters node need not have a fixed filter pipeline, it is possible to change that filter pipeline completely at runtime.

moratom

Thanks Jakub!

Left some comments&suggestions, generally looks good.
Let's also add an example (can be similar to stereo_depth_from_host.py) and tests before merging.

moratom · 2025-04-28T12:37:59Z

src/pipeline/datatype/ImgFrame.cpp

@@ -187,6 +187,23 @@ ImgFrame& ImgFrame::setMetadata(const std::shared_ptr<ImgFrame>& sourceFrame) {
    return setMetadata(*sourceFrame);
 }

+ImgFrame& ImgFrame::setDataFrom(const ImgFrame& sourceFrame) {


I think we should go with copyDataFrom to make it clearer it's not a move.

include/depthai/pipeline/node/host/DepthFilters.hpp

moratom · 2025-04-28T13:20:44Z

include/depthai/properties/DepthFiltersProperties.hpp

+typedef dai::StereoDepthConfig::PostProcessing::SpatialFilter SpatialFilterParams;
+typedef dai::StereoDepthConfig::PostProcessing::SpeckleFilter SpeckleFilterParams;
+typedef dai::StereoDepthConfig::PostProcessing::TemporalFilter TemporalFilterParams;
+typedef std::variant<MedianFilterParams, SpatialFilterParams, SpeckleFilterParams, TemporalFilterParams> FilterParams;


Probbably better to move to the filters to a common location and then use using ... in both StereoDepth and here

moratom · 2025-04-28T13:30:25Z

include/depthai/pipeline/datatype/DepthFiltersConfig.hpp

+    /**
+     * Index of the filter to be applied
+     */
+    std::int32_t filterIndex;
+
+    /**
+     * Parameters of the filter to be applied
+     */
+    FilterParams filterParams;
+


Any reason why we didn't go with std::vector here?

The idea is as follows: Say we have filters, applied sequentially, (F1 | F2 | F3 | F4 | F5). When one wants to modify the properties any one filter, say F3, one can simply specify its index and properties without modifying or having to worry about all the other filters. This will throw an error if the param's type does not match the expected type of the filter at the given index.

moratom · 2025-04-28T13:31:25Z

include/depthai/properties/DepthFiltersProperties.hpp

+struct SequentialDepthFiltersProperties : PropertiesSerializable<Properties, SequentialDepthFiltersProperties> {
+    /**
+     * List of filters (the type of which is determined by the filter parameters) to apply to the input frame
+     */
+    std::vector<FilterParams> filters;
+};
+


Let's rather use an initialConfig, to not duplicate parameters.

Read this, the properties and the config are different, we have a fixed sequence of filters, each of which one can set the properties at runtime for.

moratom · 2025-04-28T13:32:44Z

include/depthai/properties/DepthFiltersProperties.hpp

+struct DepthConfidenceFilterProperties : PropertiesSerializable<Properties, DepthConfidenceFilterProperties> {
+    /**
+     * Threshold for the confidence filter
+     */
+    float confidenceThreshold = 0.0f;
+};


Same as for DepthFilter properties

src/pipeline/node/host/DepthFilters.cpp

SzabolcsGergely · 2025-04-30T21:40:28Z

src/pipeline/node/host/DepthFilters.cpp

+
+constexpr const size_t PERSISTENCY_LUT_SIZE = 256;
+
+struct TemporalFilterParams {


why not break out each implementation in their own .cpp ?

SzabolcsGergely · 2025-04-30T21:44:22Z

src/pipeline/node/host/DepthFilters.cpp

@@ -0,0 +1,904 @@
+#include "depthai/pipeline/node/host/DepthFilters.hpp"


i'd rather call these ImageFilters, they can be used for general purposes, if one wants to.
Only depthConfidence filter is ToF specific?
E.g. temporal filter could be used on input images to stereo depth to reduce noise. Same for median.

Correct, the DepthConfidenceFilter is specifically designed to used alongside the ToF node.

Copilot

Pull Request Overview

Adds two new host-runnable filter nodes (image filters and depth-confidence filter) to the pipeline, along with their data/config types, properties, bindings, and example scripts.

Introduce ImgFrame::copyDataFrom and clone methods to duplicate frames
Define ImageFilters and DepthConfidenceFilter nodes, their configs, properties, and Python/JS bindings
Refactor StereoDepthConfig to use centralized FilterParams, update datatype enums, CMake, and examples

Reviewed Changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/pipeline/datatype/ImgFrame.cpp	Implement `copyDataFrom` and `clone` for frame duplication
include/depthai/pipeline/datatype/ImgFrame.hpp	Declare `copyDataFrom`/`clone` and update doc comments
src/pipeline/datatype/DatatypeEnum.cpp	Add new datatypes to hierarchy for image/depth-confidence
include/depthai/pipeline/datatype/DatatypeEnum.hpp	Extend `DatatypeEnum` with new config entries
include/depthai/properties/ImageFiltersProperties.hpp	Define properties for both filter nodes
include/depthai/pipeline/node/ImageFilters.hpp	Create `ImageFilters` and `DepthConfidenceFilter` nodes
include/depthai/pipeline/datatype/StereoDepthConfig.hpp	Refactor post-processing filters to `FilterParams`
include/depthai/pipeline/FilterParams.hpp	Introduce centralized filter parameter structs/enums
include/depthai/pipeline/datatype/ImageFiltersConfig.hpp	New buffer messages for runtime filter reconfiguration
bindings/python/.../FilterParamsBindings.*	Python bindings for filter parameters
bindings/python/.../ImageFiltersBindings.cpp	Python/node bindings for new filter nodes
bindings/js/bindings.cpp	Add `ImageFiltersConfig` to JS bindings
CMakeLists.txt	Link OpenCV calib3d, include new source and binding files
examples/python/.../stereo_depth_filters.py	Python example for image filters on stereo disparity
examples/python/RVC2/ToF/tof_host_filter.py	Python example for depth-confidence filter with ToF node

Comments suppressed due to low confidence (1)

src/pipeline/datatype/ImgFrame.cpp:200

New clone method should have unit tests verifying deep copy of both metadata and pixel data.

std::shared_ptr<ImgFrame> ImgFrame::clone() const {

Copilot · 2025-05-25T00:35:43Z

include/depthai/pipeline/datatype/ImgFrame.hpp

@@ -310,6 +310,25 @@ class ImgFrame : public Buffer, public ProtoSerializable {
     */
    ImgFrame& setMetadata(const std::shared_ptr<ImgFrame>& sourceFrame);

+    /**
+     * Convience function to set the data of the ImgFrame


Typo in doc comment: 'Convience' should be 'Convenience'.

include/depthai/pipeline/datatype/ImgFrame.hpp

src/pipeline/datatype/ImgFrame.cpp

…[skip ci]

…Camera node [skip ci]

…ereo_filters

include/depthai/pipeline/datatype/ImageFiltersConfig.hpp

include/depthai/properties/ImageFiltersProperties.hpp

moratom · 2025-06-19T15:41:39Z

src/pipeline/node/ImageFilters.cpp

Maybe it would be worth splitting this into files per filter and removing redundant constructors and other warnings brought up by clangd.

lnotspotl · 2025-07-08T20:59:24Z

@SzabolcsGergely This should be, hopefully, the final version of this PR and only minor changes will be made. Could you go over the PR one more time, pointing out deficiencies and potential improvement points?

We will also have @DavidZah (or someone else from the CV team) add in preset-modes for the filter pipelines and test the overall implementation.

src/pipeline/node/ImageFilters.cpp

lnotspotl added 4 commits April 28, 2025 00:38

Initial depth filter host node implementation

b102b01

Enable dynamic filter property setting

9d02266

Dynamic depth confidence node config

d45e14c

Dynamic depth confidence node config

6e44920

lnotspotl self-assigned this Apr 28, 2025

lnotspotl requested a review from moratom April 28, 2025 06:28

moratom reviewed Apr 28, 2025

View reviewed changes

SzabolcsGergely reviewed Apr 30, 2025

View reviewed changes

lnotspotl added 6 commits May 6, 2025 03:54

Update naming, fix ToF filter

373b2a7

Move depth filters out of host node folder

af39e67

follow camelCase naming

2d18856

Merge branch 'v3_develop' into feature/extract_stereo_filters

9e29e10

Update variable names

3dfc5f9

remove unused variables

d8983c9

lnotspotl marked this pull request as draft May 6, 2025 08:51

lnotspotl added 9 commits May 6, 2025 04:58

follow camelCase naming convention

e236d5b

follow camelCase naming convention

b89c59c

Rename DepthFilters to ImageFilters

a0dbb4f

extract commonalities between image filters and stereo depth node

c1ec546

Update docstrings

1b53532

Update docstrings

79c4e51

add filter host node example

22648d5

fix temporal filter for different ImgFrame types

13435b0

Remove redundant printf

81c3436

lnotspotl requested review from SzabolcsGergely and moratom May 6, 2025 13:07

lnotspotl marked this pull request as ready for review May 6, 2025 14:43

lnotspotl added 3 commits May 7, 2025 04:39

fix python docs generation

8357c13

fix docs build

ad0e426

Update ImageFilters.cpp

dbccf99

lnotspotl added 4 commits May 13, 2025 15:59

Update ImageFilters.cpp

bd1d73c

use smart pointers, use depthai checks instead of throws

29d69e9

fix confidence

3f1384a

fix confidence

c7f23b2

SzabolcsGergely requested a review from Copilot May 25, 2025 00:34

Copilot AI reviewed May 25, 2025

View reviewed changes

lnotspotl added 6 commits June 9, 2025 09:22

merge v3_develop in [skip ci]

02bbfa0

Fix errors, rename DepthConfidenceFilter to ToFDepthConfidenceFilter …

00fafff

…[skip ci]

add preset modes, fix warnings and errors [skip ci]

38d2598

update examples to use the Camera node instead of the depracated Mono…

97525dd

…Camera node [skip ci]

Merge branch 'develop' into feature/extract_stereo_filters

05d64b4

merge develop in, update ToF host filters example

58ad706

lnotspotl changed the base branch from v3_develop to develop June 16, 2025 19:50

Merge remote-tracking branch 'origin/develop' into feature/extract_st…

f8ff12f

…ereo_filters

moratom reviewed Jun 19, 2025

View reviewed changes

include/depthai/pipeline/datatype/ImageFiltersConfig.hpp Outdated Show resolved Hide resolved

moratom reviewed Jun 19, 2025

View reviewed changes

Matevz Morato and others added 7 commits June 19, 2025 17:43

Formatting

d152967

Merge branch 'develop' into feature/extract_stereo_filters

6211457

Update configs and properties

ef2024f

[skip ci] update default value for ToF's confidence threshold

7a70a91

Add filter preset modes

1fe468b

fix spelling

ec54c7f

ToF node enhancements

940720d

lnotspotl changed the title ~~Feature/extract stereo filters~~ ToF node enhancements, host-runnable ToF and image filters Jul 1, 2025

lnotspotl added 3 commits July 1, 2025 19:51

Merge branch 'develop' into feature/extract_stereo_filters

af849e7

Merge branch 'develop' into feature/extract_stereo_filters

b47a4ab

improve image filter logging

2f79d29

SzabolcsGergely reviewed Jul 9, 2025

View reviewed changes

src/pipeline/node/ImageFilters.cpp Outdated Show resolved Hide resolved

fix wrong wiring

39b30aa


		constexpr const size_t PERSISTENCY_LUT_SIZE = 256;

		struct TemporalFilterParams {

		@@ -0,0 +1,904 @@
		#include "depthai/pipeline/node/host/DepthFilters.hpp"

ToF node enhancements, host-runnable ToF and image filters #1308

Are you sure you want to change the base?

ToF node enhancements, host-runnable ToF and image filters #1308

Uh oh!

Conversation

lnotspotl commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

moratom left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI May 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lnotspotl commented Jul 8, 2025

Uh oh!

Uh oh!

Uh oh!

lnotspotl commented Apr 28, 2025 •

edited

Loading