Add Mean Average Precision (mAP) metric

[ternaus]

The main metric for object detection tasks is the Mean Average Precision, implemented in PyTorch, and computed on GPU.

It would be nice to add it to the collection of the metrics.

The example implementation using numpy:

https://github.com/ternaus/iglovikov_helper_functions/blob/master/iglovikov_helper_functions/metrics/map.py

[OrthoDex]

Hello! First time contributor here. I'd like to take a shot at this!

[awaelchli]

would be great to have that metric!

[NumesSanguis]

Not yet sure yet, but it looks like AveragePrecision (https://pytorch-lightning.readthedocs.io/en/stable/metrics.html) seems to be acting like mAP. but they didn't add "Mean" to the name?

[Kshitij09]

I recently managed to implement a minimal version of Kaggle's Mean Average Precision metric. The difference is in the calculation itself. You can find the details here:

https://www.kaggle.com/c/global-wheat-detection/overview/evaluation

My kernel: pytorch-mean-absolute-precision-calculation

With slight modifications in the formula of mAP, I suppose we would be able to integrate this metric in pytorch-lightning (since we already have the average-precision implemented)

I've already written a logic to map predicted boxes to ground truth ones (taking their respective scores into consideration) so have a look at the kernel and let me know if you found any issues.

[briankosw]

Hi, I would love to take this on. @SkafteNicki could you assign it to me?

[SkafteNicki]

@briankosw thanks you for wanting to contribute!
Please ping in your PR.

[Vichoko]

We need this! 🤩

For PyTorch lightning modules focused on multi-class classification, would be useful to obtain mAP metric over the test_dataloader data samples in a standard way that works with DDP with minor changes to the architecture. Because this metric is commonly used to conclude about the representation learning side of the deep learning architectures without using any thresholds or max functions to take hard decisions on the predictions.

Most classifiers have a pair of Fully-connected layers that do a classwise likelihood prediction. The output of the last layer before the FC layers could be used for a retrieval task over the test dataset and mAP could be calculated over this.

[jspaezp]

@Borda I can give this a go.

But I am wondering what input format would be the most consistent with the rest of the package ...
Most of the stuff that it implemented in the package takes directly tensors of shape [N,C, ...] ... and what scope would you like for it ...

So directly ...

1. Should it support only pre-aligned inputs? EDIT: I don't think so ... but supporting box inputs requires this to be implemented
   • (preds = [N,C] , targets = [N,C] )
2. Should it support bounding boxes ? if so in what format (xxyy, xyxy, xwyh, xyhw ...)? Should each update be for a single "group" (each image for instance) or should it take a grouping argument?
   • (preds: float = [N1,C] , targets: long/bool = [N2,C] , pred_boxes = [x,x,y,y], target_boxes = [x,x,y,y], pred_grouping: long = [N1], target_grouping: long = [N2], iou_cutoff = 0.5); where the values in the grouping are in the range [0, NUM_GROUPS] so only boxes in the same group are compared with each other.
3. Should it support masks?
   • Option 1: (preds: float = [N,C] , targets: long/bool = [N,C] , pred_masks = [N,H,W], target_masks = [N,H,W], grouping = [N], iou_cutoff = 0.5, mask_threshold = 0.5)
   • Option 2: (preds: float = [N1,C,H,W] , targets: long/bool = [N2,C,H,W], pred_grouping: long = [N1], target_grouping: long = [N2], iou_cutoff = 0.5, mask_threshold = 0.5)

On the other hand ... the COCO implementation has for each image a set of positive and negative labels in each image and ignores the rest ... should that be supported?

lmk what you think
Kindest wishes,
Sebastian

edit: leaving here for reference

[lucadiliello]

mAP was added to PL some months ago and is now part of the torchmetrics package.

[Borda]

added with 60c0eec