Add bounding box sensor

[adlarkin]

Related to #134

Desired behavior

Add a bounding box sensor to Ignition.

Alternatives considered

N/A (this is a new feature)

Implementation suggestion

We will need functionality added to the following repositories:

• https://github.com/ignitionrobotics/ign-gazebo
  • add a system plugin that allows users to specify labels for objects that they'd like to generate bounding boxes for
• https://github.com/ignitionrobotics/ign-sensors
  • add the bounding box sensor that will publish bounding box data to a topic via ign-transport
• https://github.com/ignitionrobotics/ign-rendering
  • use rendering engines (like Ogre2) to actually generate bounding boxes
• https://github.com/osrf/sdformat
  • add a bounding box sensor to the SDF specification
• https://github.com/ignitionrobotics/ign-msgs
  • add a bounding box message type

Additional context

A lot of work for implementing this sensor is already underway. Here are all of the related PRs that have been opened:

• Label Component & System gz-sim#853
• Bounding Box Camera gz-rendering#334
• Add segmentation and bounding box sensor types sdformat#592

[chapulina]

It would be interesting to add some more context about the use case for the sensor and what kind of data it should produce. I think it's especially important to define these clearly because this sensor wouldn't be replicating any real physical sensors. In the real world, this kind of data is obtained by operating on data provided by other sensors, like cameras.

[adlarkin]

It would be interesting to add some more context about the use case for the sensor and what kind of data it should produce.

The original thought was to use this sensor to provide users with datasets that could be used for training machine learning models. Manually generating bounding boxes for images can be difficult and/or take a while, so the hope is that the process of data collection can be made easier by using this sensor with various scenes that are created in simulation.

Regarding what data it should produce: the thought is to provide users with an image and a corresponding bounding box message which provides the bounding box information for all of the requested objects in a scene (a new bounding box message type needs to be created in ign-msgs). The bounding box message can look similar to the bounding box message type used in darknet_ros. So, once a user creates a simulation world and attaches labels to objects that the user wants bounding boxes for (see gazebosim/gz-sim#853), an image would be captured of the scene and bounding box data would be specified for the objects in the scene requested by the user (assuming that the objects are visible, of course). To take this a step further, the scene could be altered (changing the camera position, object positions, lighting, etc.) to generate another image with bounding box data. If we add a way to programmatically alter the scene, this would allow users to quickly generate hundreds of various scenes with corresponding bounding box data.

this sensor wouldn't be replicating any real physical sensors. In the real world, this kind of data is obtained by operating on data provided by other sensors, like cameras.

Yep 👍 this sensor is meant to provide a way for automatic bounding box generation so that users don't need to operate on data from things like cameras. As mentioned earlier in my comment, the main use case for this would probably be dataset generation for ML models, but perhaps users will find this useful for other use cases as well.

---

@chapulina does that help answer your questions and/or clarify the design/motivation behind this sensor? Or is there anything else that needs to be addressed/considered?

[chapulina]

Thank you for all the clarification, this is great context 👍

The bounding box message can look similar to the bounding box message type used in darknet_ros.

As a reference, I'll also link to vision_msgs, which provides some different bounding box messages.

---

Just some clarifications about the bounding box data:

• Will the boxes be always aligned?
• Will the boxes be 2D, 3D or both?

[adlarkin]

Will the boxes be always aligned?

Yes, I believe so - are there reasons/use cases that should make us consider using bounding boxes that aren't axis aligned?

Will the boxes be 2D, 3D or both?

Right now, development is being done for 2D bounding boxes only. However, I am sure that 3D bounding boxes would also be useful, so perhaps we can add in 3D functionality once the 2D implementation has been completed.

[chapulina]

are there reasons/use cases that should make us consider using bounding boxes that aren't axis aligned?

I'm not sure, but bounding box messages in ROS are oriented, so I think it's worth documenting the rationale for orienting it or not. As a reference, here's some discussion on the vision message standards for ROS that could be useful. The vision_msgs README also has useful information.

perhaps we can add in 3D functionality once the 2D implementation has been completed

My only concern is not to have to go back and change messages and APIs if we decide to support 3D later. If we want to leave the way open for it, we may want to use 3D types on the APIs for future-proofing.

[AmrElsersy]

The bounding boxes in 3D datasets are always oriented boxes, but in 2D datasets are usually axis aligned.
the 3d boxes format in datasets is usually
center(x,y,z), width, height, depth, orientation just like the ROS msgs you mentioned Louise

My only concern is not to have to go back and change messages and APIs if we decide to support 3D later. If we want to leave the way open for it, we may want to use 3D types on the APIs for future-proofing.

Ok, I will work on that, on providing 3D boxes in rendering first, then will modify all the PRs of the bounding box stuff with the new format, which is the oriented boxes just like the ROS msg

Is that make sense ?

[AmrElsersy]

Also another thing, should we have a separate msgs for 2D boxes and 3D boxes ?? like the ROS msgs (BoundingBox3D, BoundingBox2D), or we can use the 3D boxes format with the 2D also and say "its ok the 2D will have the same format but with z=0 and zero orientation"

.....
Also, about the label ... in the ROS msgs, the bounding boxes msgs just have the dim/rotation information, and they have a separate msg for the detection (combining the label with box) , should we do that ??
I think it is ok to just combine the label with the dim/rotation info directly, so is that ok also ?