IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

[SlowMonk]

When coco dataset has empty bbox annotation
following error occured.
How should I handle the empty bbox dataset in cocodataset

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

[{'boxes': tensor([], device='cuda:1'), 'labels': tensor([], device='cuda:1', dtype=torch.int64), 'image_id': tensor([311877], device='cuda:1'), 'area': tensor([], device='cuda:1'), 'iscrowd': tensor([], device='cuda:1', dtype=torch.int64)}]
len_> 0

class CocoDataset(torch.utils.data.Dataset):
    def __init__(self, root, annotation, transforms=None):

        self.root = root
        self.transforms = transforms
        self.coco = COCO(annotation)
        self.ids = list(sorted(self.coco.imgs.keys()))

    def __getitem__(self, index):
        # Own coco file
        coco = self.coco
        # Image ID
        img_id = self.ids[index]
        # List: get annotation id from coco
        ann_ids = coco.getAnnIds(imgIds=img_id)
        # Dictionary: target coco_annotation file for an image
        coco_annotation = coco.loadAnns(ann_ids)
        # path for input image
        path = coco.loadImgs(img_id)[0]['file_name']
        # open the input image
        img = Image.open(os.path.join(self.root, path))

        # number of objects in the image
        num_objs = len(coco_annotation)

        # Bounding boxes for objects
        # In coco format, bbox = [xmin, ymin, width, height]
        # In pytorch, the input should be [xmin, ymin, xmax, ymax]
        boxes = []
        for i in range(num_objs):
            xmin = coco_annotation[i]['bbox'][0]
            ymin = coco_annotation[i]['bbox'][1]
            xmax = xmin + coco_annotation[i]['bbox'][2]
            ymax = ymin + coco_annotation[i]['bbox'][3]
            boxes.append([xmin, ymin, xmax, ymax])
        boxes = torch.as_tensor(boxes, dtype=torch.float32)
        # Labels (In my case, I only one class: target class or background)
        labels = torch.ones((num_objs,), dtype=torch.int64)
        # Tensorise img_id
        img_id = torch.tensor([img_id])
        # Size of bbox (Rectangular)
        areas = []
        for i in range(num_objs):
            areas.append(coco_annotation[i]['area'])
        areas = torch.as_tensor(areas, dtype=torch.float32)
        # Iscrowd
        iscrowd = torch.zeros((num_objs,), dtype=torch.int64)

        # Annotation is in dictionary format
        my_annotation = {}
        my_annotation["boxes"] = boxes
        my_annotation["labels"] = labels
        my_annotation["image_id"] = img_id
        my_annotation["area"] = areas
        my_annotation["iscrowd"] = iscrowd

        if self.transforms is not None:
            img = self.transforms(img)
        return img, my_annotation


def resize_boxes(boxes, original_size, new_size):
    ratios = tuple(float(s) / float(s_orig) for s, s_orig in zip(new_size, original_size))
    ratio_height, ratio_width = ratios
    #print("=============================================================")
    #print('ratio_height->',ratio_height,'ratio_width->',ratio_width,'boxes->',boxes)
    print('len_>',len(boxes))

    xmin, ymin, xmax, ymax = boxes.unbind(1)
    xmin = xmin * ratio_width
    xmax = xmax * ratio_width
    ymin = ymin * ratio_height
    ymax = ymax * ratio_height

    return torch.stack((xmin, ymin, xmax, ymax), dim=1)

[zhangguanheng66]

@fmassa Could we add a check to avoid the out of range problem?

[fmassa]

Hi,

This is an issue with your dataset implementation.
When there are no boxes in it, the boxes are expected to be of shape 0 x 4, but instead you have a tensor of shape 0.

You can fix this by doing

boxes = torch.as_tensor(boxes, dtype=torch.float32)
boxes = boxes.reshape(-1, 4)

in order to guarantee your boxes will always have two dimensions.

But I agree with @zhangguanheng66 and we could add another check in the model to validate that the inputs have the correct shapes.

[fmassa]

I propose to add a check in generalized_rcnn.py for now, as it is the main entrypoint for those functions.

[mmcenta]

Hello, I would like to work on this issue.

[fmassa]

@mmcenta go for it!

[hakontonne]

Hello, couldn't this be done more easily than with extensive tests?
I've encountered this issue myself and fixed it locally by changing

bbox = target["boxes"]
bbox = resize_boxes(bbox, (h, w), image.shape[-2:]) #This call raises Index out of bounds, unbind on empty tensor       
target["boxes"] = bbox

in GeneralizedRCNNTransform to

bbox = target["boxes"]
bbox = resize_boxes(bbox, (h, w), image.shape[-2:]) if bbox.numel() != 0 else bbox
target["boxes"] = bbox

This is simple, fixes the issue and is consistent with the actual check for background image in roi_heads.py:

if gt_boxes_in_image.numel() == 0:
                # Background image

It's the bbox.numel() == 0 that determines if a label is background or not, then as long as a dataset implementation passes this check for background class, it should work? Would it not appropriate that the network is as agnostic to the dataset implementation as possible? If not, shouldn't the documentation be updated to reflect that?

I've checked that this works and this would be more consistent with how a background class is determined. The network will then accept empty boxes tensors and boxes tensors of size 0 x 4 and is guaranteed to work as it uses the same test as roi_heads.py.

[fmassa]

@hakontonne the issue with this approach is that we are propagating "unexpected" tensor shapes, which might work in some cases but is prone to failure.

I think it's ok to ask users to format their dataset in a way which is uniform -- providing boxes which are of shape 0 sometimes is not as uniform as if they have always Nx4 shape (with N potentially being 0).

[Mrgove10]

Facing the same issue.
I am folowing the code from this tutorial btw : https://medium.com/fullstackai/how-to-train-an-object-detector-with-your-own-coco-dataset-in-pytorch-319e7090da5

Applying the

boxes = torch.as_tensor(boxes, dtype=torch.float32)
boxes = boxes.reshape(-1, 4)

fix does lead to better results but makes a second error popup :

File "/home/adrien/.local/lib/python3.8/site-packages/torchvision/models/detection/roi_heads.py", line 760, in forward
    loss_classifier, loss_box_reg = fastrcnn_loss(
File "/home/adrien/.local/lib/python3.8/site-packages/torchvision/models/detection/roi_heads.py", line 43, in fastrcnn_loss
    box_regression = box_regression.reshape(N, -1, 4)
RuntimeError: cannot reshape tensor of 0 elements into shape [0, -1, 4] because the unspecified dimension size -1 can be any value and is ambiguous

[datumbox]

@Mrgove10 This is probably fixed by #3031. Could you try on the latest master and let me know if you still get the problem?

[Mrgove10]

@datumbox In the end i switched and went with Detectron2, not planing to retest anytime soon sorry 😞

[vadimkantorov]

@fmassa I propose resize_boxes to use dim=-1 instead of dim=1. This would allow to use this function for user tensors that are not just Bx4 but also B1xB2x4 and will allow more code reuse. I hope this method would go to a shared box_ops/boxes namespace some day (similar to detectron2)