Bug Description
In sam3/model/sam3_video_base.py, the memory encoded during reconditioning is
immediately overwritten, making reconditioning ineffective at the memory level.
Root Cause
In run_tracker_update_planning_phase, the execution order is:
_recondition_masklets() (line ~727) — calls add_new_mask() +
propagate_in_video_preflight(run_mem_encoder=True), encoding detector masks
into output_dict[frame_idx]["maskmem_features"]
_tracker_update_memories() (line ~757) — re-runs memory encoder using
tracker_low_res_masks_global (the original tracker masks, unchanged by
reconditioning), and overwrites the same maskmem_features for the same
frame_idx
The overwrite happens here in _tracker_update_memories:
output_dict[storage_key][frame_idx]["maskmem_features"] = local_maskmem_features
output_dict[storage_key][frame_idx]["maskmem_pos_enc"] = [...]
Since tracker_low_res_masks_global is never updated with the reconditioned detector
masks, the memory encoder in step 2 encodes the old tracker masks, discarding the
detector mask memory from step 1.
Impact
- Reconditioned detector masks are correctly used for output (via
_execute_tracker_update_phase), but their memory is lost
- Subsequent frames attend to memory encoded from old tracker masks, not the
corrected detector masks
- Both periodic (recondition_every_nth_frame) and IoU-based reconditioning are
affected
Suggested Fix
Update tracker_low_res_masks_global with reconditioned detector masks before calling
_tracker_update_memories(), so a single memory encoding pass uses the correct
masks:
Before _tracker_update_memories, replace masks for reconditioned objects
if reconditioned_obj_ids:
for trk_obj_id in reconditioned_obj_ids:
det_idx = trk_id_to_max_iou_high_conf_det.get(trk_obj_id)
if det_idx is None:
continue
trk_idx = np.where(tracker_metadata_prev["obj_ids_all_gpu"] ==
trk_obj_id)[0].item()
tracker_low_res_masks_global[trk_idx] = det_out["mask"][det_idx]
Then remove run_mem_encoder=True from _recondition_masklets
Affected Versions
Verified present from Initial commit (a13e358) through latest (2e0009e v3 code).
Bug Description
In
sam3/model/sam3_video_base.py, the memory encoded during reconditioning isimmediately overwritten, making reconditioning ineffective at the memory level.
Root Cause
In
run_tracker_update_planning_phase, the execution order is:_recondition_masklets()(line ~727) — callsadd_new_mask()+propagate_in_video_preflight(run_mem_encoder=True), encoding detector masksinto
output_dict[frame_idx]["maskmem_features"]_tracker_update_memories()(line ~757) — re-runs memory encoder usingtracker_low_res_masks_global(the original tracker masks, unchanged byreconditioning), and overwrites the same
maskmem_featuresfor the sameframe_idxThe overwrite happens here in
_tracker_update_memories:Impact
_execute_tracker_update_phase), but their memory is lost
corrected detector masks
affected
Suggested Fix
Update tracker_low_res_masks_global with reconditioned detector masks before calling
_tracker_update_memories(), so a single memory encoding pass uses the correct
masks:
Before _tracker_update_memories, replace masks for reconditioned objects
Then remove run_mem_encoder=True from _recondition_masklets
Affected Versions
Verified present from Initial commit (a13e358) through latest (2e0009e v3 code).