Skip to content

recondition_masklets encoded memory is overwritten by _tracker_update_memories #531

@amier2775

Description

@amier2775

Bug Description

In sam3/model/sam3_video_base.py, the memory encoded during reconditioning is
immediately overwritten, making reconditioning ineffective at the memory level.

Root Cause

In run_tracker_update_planning_phase, the execution order is:

  1. _recondition_masklets() (line ~727) — calls add_new_mask() +
    propagate_in_video_preflight(run_mem_encoder=True), encoding detector masks
    into output_dict[frame_idx]["maskmem_features"]
  2. _tracker_update_memories() (line ~757) — re-runs memory encoder using
    tracker_low_res_masks_global (the original tracker masks, unchanged by
    reconditioning), and overwrites the same maskmem_features for the same
    frame_idx

The overwrite happens here in _tracker_update_memories:

output_dict[storage_key][frame_idx]["maskmem_features"] = local_maskmem_features
output_dict[storage_key][frame_idx]["maskmem_pos_enc"] = [...]

Since tracker_low_res_masks_global is never updated with the reconditioned detector
masks, the memory encoder in step 2 encodes the old tracker masks, discarding the
detector mask memory from step 1.

Impact

  • Reconditioned detector masks are correctly used for output (via
    _execute_tracker_update_phase), but their memory is lost
  • Subsequent frames attend to memory encoded from old tracker masks, not the
    corrected detector masks
  • Both periodic (recondition_every_nth_frame) and IoU-based reconditioning are
    affected

Suggested Fix

Update tracker_low_res_masks_global with reconditioned detector masks before calling
_tracker_update_memories(), so a single memory encoding pass uses the correct
masks:

Before _tracker_update_memories, replace masks for reconditioned objects

if reconditioned_obj_ids:
    for trk_obj_id in reconditioned_obj_ids:
        det_idx = trk_id_to_max_iou_high_conf_det.get(trk_obj_id)
        if det_idx is None:
            continue
        trk_idx = np.where(tracker_metadata_prev["obj_ids_all_gpu"] ==
trk_obj_id)[0].item()
        tracker_low_res_masks_global[trk_idx] = det_out["mask"][det_idx]

Then remove run_mem_encoder=True from _recondition_masklets

Affected Versions

Verified present from Initial commit (a13e358) through latest (2e0009e v3 code).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions