Memory problem with GOTURN tracker

[ieliz]

System information (version)

• OpenCV => 4.3.0
• Operating System / Platform => Windows 10 / 64, Ubuntu 18.04 / 64
• Compiler => Visual Studio 2019, gcc 7.5.0

Detailed description

Hello! I tried to use GOTURN tracker and face the problem with memory:
if tracker lose the object or even see some sort of hard obstacles, tracker starting to use too much memory.
I saw that issues only on several videos in the LaSOT dataset`s testing part, for example - 'coin-3' and 'airplane-15' videos.
For detecting the problem I tried to use psutil module for Python 3, but when tracker starts using too much memory and the module doesn't work properly and does not show anything. So, for detecting the problem, I just looked at Task Manager in Win 10 and saw how tracker starts to use 100% of the memory and after a few seconds, PC was frozen. In Ubuntu process is just killed after few seconds.
As I understand, this issue is about that, but it is closed without some kind of direct solution.
Link to archive with 'coin-3' video: https://drive.google.com/file/d/12YJnAc9Hbf45aT7mbFsrmBfdylC1WGOR/view?usp=sharing.
Link to GOTURN model: https://drive.google.com/file/d/1OARDpmA6zdpuAfQkdLAbrMHDF9PVJ3l_/view?usp=sharing.
Link to prototxt file: https://drive.google.com/file/d/1n02YeFW5_usZzPfXws4LvsVbsj1_TMSk/view?usp=sharing.
Also, here are similar questions from the answers.opencv.org :

1. https://answers.opencv.org/question/194701/goturn-two-questions/
2. https://answers.opencv.org/question/223032/object-tracking-goturn-failed-to-allocate-90-gb-cvoutofmemoryerror/

Steps to reproduce

Python 3 script for reproducing:

import cv2 as cv
import numpy as np
import argparse
import os

def get_iou(new, gt):
    new_xmin, new_ymin, new_w, new_h = new
    gt_xmin, gt_ymin, gt_w, gt_h = gt
    def get_max_coord(coord, size): return coord + size - 1.0
    new_xmax, new_ymax = get_max_coord(new_xmin, new_w), get_max_coord(
        new_ymin, new_h)
    gt_xmax, gt_ymax = get_max_coord(gt_xmin, gt_w), get_max_coord(
        gt_ymin, gt_h)
    dx = max(0, min(new_xmax, gt_xmax) - max(new_xmin, gt_xmin))
    dy = max(0, min(new_ymax, gt_ymax) - max(new_ymin, gt_ymin))
    area_of_overlap = dx * dy
    area_of_union = (new_xmax - new_xmin) * (new_ymax - new_ymin) + (
        gt_xmax - gt_xmin) * (gt_ymax - gt_ymin) - area_of_overlap
    iou = area_of_overlap / area_of_union if area_of_union != 0 else 0
    return iou

def main():
    parser = argparse.ArgumentParser(
        description='Run reproducer of the issue with GOTURN tracker')
    parser.add_argument(
        '--input', type=str, help='Full path to the folder with input source')
    parser.add_argument("--v", dest="visualization", action='store_true',
                    help="Showing process of tracking")
    args = parser.parse_args()

    tracker_name = 'GOTURN'
    video_name = 'coin-3'

    print(tracker_name, '\n', video_name)

    gt_file = open(os.path.join(args.input, video_name, "groundtruth.txt"), "r")
    gt_bb = gt_file.readline().rstrip("\n").split(",")
    init_bb = tuple([float(b) for b in gt_bb])
    video_sequence = sorted(os.listdir(os.path.join(
        args.input, video_name, "img")))

    tracker = cv.TrackerGOTURN_create()
    init_once = False

    # Loop for every frame in the video
    for number_of_the_frame, image in enumerate(video_sequence):
        frame = cv.imread(os.path.join(args.input, video_name, "img", image))
        gt_bb = tuple([float(x) for x in gt_bb])


        # Check for presence of the object on the image
        # Image is ignored if no object on it
        if gt_bb[2] == 0 or gt_bb[3] == 0:
            gt_bb = gt_file.readline().rstrip("\n").split(",")
            continue

        # Condition for reinitialization of the tracker
        if ((number_of_the_frame + 1) % 250 == 0):
            tracker = cv.TrackerGOTURN_create()
            init_once = False
            init_bb = gt_bb

        if not init_once:
            init_state = tracker.init(frame, init_bb)
            init_once = True

        init_state, new_bb = tracker.update(frame)

        # Visualization of the tracking process if needed
        if args.visualization:
            new_x, new_y, new_w, new_h = list(map(int, new_bb))
            cv.rectangle(frame, (new_x, new_y), ((
                new_x + new_w), (new_y + new_h)), (200, 0, 0))
            cv.imshow("GOTURN tracking process", frame)
            cv.waitKey(1)

        # Intersection over Union shows how the behaviour of the tracker
        # is changing with every frame
        iou = get_iou(new_bb, gt_bb)
        print('Number of the frame = ', number_of_the_frame, ' IoU = ', iou)

if __name__ == '__main__':
    main()

Issue submission checklist

• I report the issue, it's not a question
• I checked the problem with documentation, FAQ, open issues,
  answers.opencv.org, Stack Overflow, etc and have not found solution
• I updated to latest OpenCV version and the issue is still there
• There is reproducer code and related data files: videos, images, onnx, etc

[dkurt]

Can reproduce the problem. May I ask to run a tracker on this video and show how it will track between after 190th frame with the following changes. It's interesting to see the dynamics how it started grow beyond the edges.

--- a/modules/tracking/src/gtrTracker.cpp
+++ b/modules/tracking/src/gtrTracker.cpp
@@ -137,11 +137,18 @@ bool TrackerGOTURNImpl::updateImpl(const Mat& image, Rect2d& boundingBox)
     targetPatchRect.x = (float)(prevCenter.x - prevBB.width*padTargetPatch / 2.0 + targetPatchRect.width);
     targetPatchRect.y = (float)(prevCenter.y - prevBB.height*padTargetPatch / 2.0 + targetPatchRect.height);
 
+    targetPatchRect.width = std::min(targetPatchRect.width, (float)prevFrame.cols);
+    targetPatchRect.height = std::min(targetPatchRect.height, (float)prevFrame.rows);
+    targetPatchRect.x = std::max(-prevFrame.cols * 0.5f, std::min(targetPatchRect.x, prevFrame.cols * 1.5f));
+    targetPatchRect.y = std::max(-prevFrame.rows * 0.5f, std::min(targetPatchRect.y, prevFrame.rows * 1.5f));

[ieliz]

I added these lines, use 'make -j8' command in build_dir, and the problem is still here.
It all starts on frame 216 - IoU starts falling. I can not see the visualization, because I am working on a remote desktop, I am sorry. Maybe I did something wrong?

UPD: Sorry, my bad. It is actually working. IoU is really low when it is reached frame 220 (equal 0 on the frame 225), but after initialization on the 250th frame, it starts tracking with a good accuracy again. I will try to figure out your solution. Thank you for your help!

[ieliz]

I tried to use this fix to measure GOTURN with the benchmark from PR#2516 and faced a new problem: tracker trying to get the ROI out of the image plane on some of LaSOT`s videos. I will think about how to fix it and for now, leave this issue open if you do not mind.

[ieliz]

I think this issue can be closed - the ROI problem is not related to the GOTURN tracker. @dkurt solution is working for the tracker (tested on the LaSOT dataset videos).

[dkurt]

if tracker lose the object or even see some sort of hard obstacles, tracker starting to use too much memory.

Issue is not solved.