More rotated bboxes transforms

[AntoineSimoulin]

Add Transforms support for Rotated Boxes

This PR is the first of a series to add support for rotated boxes transforms. In particular this PR implements the following functionalities:

• Delete the function is_rotated_bounding_box_format in "/transforms/v2/functional/_meta.py" to rely on the utility function tv_tensors.is_rotated_bounding_format introduced in Rotated bboxes transforms #9084;
• Add support for resize for rotated boxes. This function can typically transform rotated boxes into non-rectangle parallelogram. We introduce a _parallelogram_to_bounding_boxes to adjust the output box in such cases (c.f. illustration below);
• Add support for affine for rotated boxes;
• Add support for pad for rotated boxes;
• Add support for crop for rotated boxes.

Figure 1. Illustration of transformation introducing non-parallelism for rotated boxes and of how the _parallelogram_to_bounding_boxes reconstruct a rotated box from a parallelogram shape.

Test plan

Please run the following tests:

pytest test/test_transforms_v2.py -vvv -k "TestResize and test_kernel_bounding_boxes"
pytest test/test_transforms_v2.py -vvv -k "TestResize and test_bounding_boxes_correctness"

pytest test/test_transforms_v2.py -vvv -k "TestAffine and test_kernel_bounding_boxes"
pytest test/test_transforms_v2.py -vvv -k "TestAffine and test_functional_bounding_boxes_correctness"

pytest test/test_transforms_v2.py -vvv -k "TestPad and test_kernel_bounding_boxes"
pytest test/test_transforms_v2.py -vvv -k "TestPad and test_bounding_boxes_correctness"

pytest test/test_transforms_v2.py -vvv -k "TestCrop and test_kernel_bounding_box"
pytest test/test_transforms_v2.py -vvv -k "TestCrop and test_functional_bounding_box_correctness"
pytest test/test_transforms_v2.py -vvv -k "TestCrop and test_transform_bounding_boxes_correctness"
pytest test/test_transforms_v2.py -vvv -k "TestCenterCrop and test_kernel_bounding_boxes"
pytest test/test_transforms_v2.py -vvv -k "TestCenterCrop and test_bounding_boxes_correctness"
pytest test/test_transforms_v2.py -vvv -k "TestResizedCrop and test_functional_bounding_boxes_correctness"

Questions

Please note that this PR introduces a change in the API signature of the resize_bounding_boxes function to add format: tv_tensors.BoundingBoxFormat and specialize the operation given whether the box is rotated or not. Contrary to other transforms, this function did typically not received the format parameter as input before. Is it possible to add it here? Should we maybe add a default value in this case to preserve the behavior for non-rotated box by default?

Future work

This PR implements only 4 transforms for rotated boxes. We plan to release the remaining 3 transforms in a subsequent PR.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/9095

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 2f322f0 with merge base 428a54c ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[NicolasHug]

Thanks @AntoineSimoulin , I have a few comments and questions but this looks good!

[NicolasHug]

Can you help me understand why we need a flip parameter in reference_affine_rotated_bounding_boxes_helper, but not in reference_affine_bounding_boxes_helper?

It seems that reference_affine_bounding_boxes_helper, the flip is done through the affine_matrix? Is this something we can do for the rotated case as well? If not it might be worth adding a comment to explain why

[AntoineSimoulin]

@NicolasHug, this is a good question. As illustrated in the image below. When we apply the flip operation, we actually transform each points (x1, y1), (x2, y2), (x3, y3), (x4, y4) from the bounding box. However the flip operation will typically change the order of the points if we go through them in clock-wise order. Given the definition of "XYXYXYXY", "CXCYWHR", and "XYWHR", we do want to go through (x1, y1), (x2, y2), (x3, y3), (x4, y4) in clock-wise order. This is why we are switching the point (x2, y2) and (x4, y4). This typically does not append for non-rotated boxes as we re-assign the points with min and max operations here. I hope it makes sense?

[AntoineSimoulin]

Thanks for the in-depth review @NicolasHug. I submitted a few commits which should answer all your comments. Here is a list of the modifications I made to address your comments:

• Modify the behavior of the _parallelogram_to_bounding_boxes function so that the first point (x1, y1) and the third point (x3, y3) of the parallelogram do not necessarily remain fixed. In this new version, the two diagonally opposed points of the parallelogram forming the longest diagonal remain fixed. The other points are adjusted to form a proper rectangle;
• Make _parallelogram_to_bounding_boxes by default not inplace and remove inplace=True argument;
• Added some testing for the _parallelogram_to_bounding_boxes function in the "/vision/test/test_transforms_v2.py" file. Please run with pytest test/test_transforms_v2.py -vvv -k "test_parallelogram_to_bounding_boxes";
• Add default value for parameter format in the resize_bounding_boxes function and set this parameter as the last for the function to ensure backward compatibility;
• Adjust tolerance for TestCrop by using torch.testing.assert_close instead of assert_equal(actual, expected, atol=1, rtol=0);
• Modify how test bounding boxes are created in the "//vision/test/common_utils.py". In the case of int rotated bounding boxes, the rounding errors induced by the casting from int to float can compound as we create each of the 4 vertices. To avoid this we do round the coordinates after the creation of each vertex.

[NicolasHug]

Thanks a lot @AntoineSimoulin !

[github-actions]

Hey @NicolasHug!

You merged this PR, but no labels were added.
The list of valid labels is available at https://github.com/pytorch/vision/blob/main/.github/process_commit.py