Add SAM 3.1

docs/source/en/_toctree.yml

@@ -1314,6 +1314,10 @@
         title: Qwen3VL
       - local: model_doc/qwen3_vl_moe
         title: Qwen3VLMoe
+      - local: model_doc/sam3_1
+        title: SAM 3.1
+      - local: model_doc/sam3_1_video
+        title: SAM 3.1 Video
       - local: model_doc/sam3
         title: SAM3
       - local: model_doc/sam3_video

docs/source/en/model_doc/sam3_1.md

@@ -0,0 +1,125 @@
+<!--Copyright 2026 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+*This model was released on {release_date} and added to Hugging Face Transformers on 2026-03-30.*
+# SAM 3.1
+
+<div style="float: right;">
+    <div class="flex flex-wrap space-x-1">
+        <img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-DE3412?style=flat&logo=pytorch&logoColor=white">
+        <img alt="SDPA" src="https://img.shields.io/badge/SDPA-DE3412?style=flat&logo=pytorch&logoColor=white">
+        <img alt="FlashAttention" src="https://img.shields.io/badge/%E2%9A%A1%EF%B8%8E%20FlashAttention-eae0c8?style=flat">
+    </div>
+</div>
+
+## Overview
+
+SAM 3.1 is the public checkpoint refresh that accompanies Meta's multiplex video-tracking release. In Transformers, image support for SAM 3.1 reuses the existing [`Sam3Model`] detector stack: we extract the detector portion of the public `facebook/sam3.1` checkpoint and load it into the standard SAM 3 image model classes.
+
+This means:
+
+- use [`Sam3Model`] and [`Sam3Processor`] for image inference;
+- convert the public `sam3.1_multiplex.pt` checkpoint before loading it with Transformers;
+- tracker-only multiplex tensors are ignored on the image path.
+
+<div class="warning">
+The public [facebook/sam3.1](https://huggingface.co/facebook/sam3.1) repository currently ships the original `sam3.1_multiplex.pt` checkpoint, not a ready-to-load Transformers weights file. Convert it first with the provided script.
+</div>
+
+## Convert the checkpoint
+
+```bash
+python src/transformers/models/sam3/convert_sam3_to_hf.py \
+  --checkpoint_path /path/to/sam3.1_multiplex.pt \
+  --output_path /path/to/sam3.1-image-hf
+```
+
+The converter auto-detects the merged SAM 3.1 checkpoint, extracts the detector weights, saves a Transformers [`Sam3Model`] repository, writes the matching processor/tokenizer files alongside it, and verifies the saved checkpoint by default. Pass `--no_verify` only if you explicitly want to skip that save/load forward check.
+
+## Usage examples with 🤗 Transformers
+
+### Text-only prompt
+
+```python
+>>> from transformers import Sam3Model, Sam3Processor
+>>> import torch
+>>> from PIL import Image
+>>> import requests
+
+>>> device = "cuda" if torch.cuda.is_available() else "cpu"
+>>> converted_path = "/path/to/sam3.1-image-hf"
+
+>>> model = Sam3Model.from_pretrained(converted_path).to(device)
+>>> processor = Sam3Processor.from_pretrained(converted_path)
+
+>>> image_url = "http://images.cocodataset.org/val2017/000000077595.jpg"
+>>> image = Image.open(requests.get(image_url, stream=True).raw).convert("RGB")
+
+>>> inputs = processor(images=image, text="ear", return_tensors="pt").to(device)
+
+>>> with torch.no_grad():
+...     outputs = model(**inputs)
+
+>>> results = processor.post_process_instance_segmentation(
+...     outputs,
+...     threshold=0.5,
+...     mask_threshold=0.5,
+...     target_sizes=inputs["original_sizes"].tolist(),
+... )[0]
+
+>>> print(outputs.pred_masks.shape)
+torch.Size([1, 200, 288, 288])
+>>> print(results["boxes"].shape)
+torch.Size([num_detections, 4])
+```
+
+### Bounding-box prompt
+
+```python
+>>> box_xyxy = [100, 150, 500, 450]
+>>> inputs = processor(
+...     images=image,
+...     input_boxes=[[box_xyxy]],
+...     input_boxes_labels=[[1]],
+...     return_tensors="pt",
+... ).to(device)
+
+>>> with torch.no_grad():
+...     outputs = model(**inputs)
+
+>>> results = processor.post_process_instance_segmentation(
+...     outputs,
+...     threshold=0.5,
+...     mask_threshold=0.5,
+...     target_sizes=inputs["original_sizes"].tolist(),
+... )[0]
+```
+
+## Sam3Config
+
+[[autodoc]] Sam3Config
+
+## Sam3Processor
+
+[[autodoc]] Sam3Processor
+    - __call__
+    - post_process_instance_segmentation
+
+## Sam3Model
+
+[[autodoc]] Sam3Model
+    - forward
+    - get_text_features
+    - get_vision_features

docs/source/en/model_doc/sam3_1_video.md

@@ -0,0 +1,120 @@
+<!--Copyright 2026 The HuggingFace Team. All rights reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
+an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.
+
+⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be
+rendered properly in your Markdown viewer.
+
+-->
+*This model was released on {release_date} and added to Hugging Face Transformers on 2026-03-30.*
+# SAM 3.1 Video
+
+<div style="float: right;">
+    <div class="flex flex-wrap space-x-1">
+        <img alt="PyTorch" src="https://img.shields.io/badge/PyTorch-DE3412?style=flat&logo=pytorch&logoColor=white">
+        <img alt="SDPA" src="https://img.shields.io/badge/SDPA-DE3412?style=flat&logo=pytorch&logoColor=white">
+        <img alt="FlashAttention" src="https://img.shields.io/badge/%E2%9A%A1%EF%B8%8E%20FlashAttention-eae0c8?style=flat">
+    </div>
+</div>
+
+## Overview
+
+SAM 3.1 introduces Meta's multiplex video-tracking architecture. In Transformers this support is exposed through [`Sam3_1VideoModel`], which loads the public `sam3.1_multiplex.pt` checkpoint into a dedicated multiplex video model implementation.
+
+Current Transformers support focuses on the public multiplex tracker checkpoint itself:
+
+- [`Sam3_1VideoModel`] provides prompted single-frame interactive decoding and multiplex propagation-head outputs;
+- the conversion script verifies parity against the upstream SAM 3.1 implementation by default;
+- the high-level `Sam3VideoProcessor` session API from [`Sam3VideoModel`] is not yet available for SAM 3.1.
+
+<div class="warning">
+The public [facebook/sam3.1](https://huggingface.co/facebook/sam3.1) repository currently ships the original `sam3.1_multiplex.pt` checkpoint. Convert it first with the provided script, then load the saved directory with [`Sam3_1VideoModel`].
+</div>
+
+## Convert the checkpoint
+
+```bash
+python src/transformers/models/sam3_1_video/convert_sam3_1_video_to_hf.py \
+  --checkpoint_path /path/to/sam3.1_multiplex.pt \
+  --output_dir /path/to/sam3.1-video-hf
+```
+
+The converter remaps the multiplex checkpoint into [`Sam3_1VideoModel`] and runs an upstream parity check by default. Pass `--no_verify` only if you explicitly want to skip that verification step.
+
+## Usage examples with 🤗 Transformers
+
+### Point-prompted frame inference
+
+```python
+>>> import torch
+>>> from PIL import Image
+>>> import requests
+>>> from transformers import Sam3ImageProcessor, Sam3_1VideoModel
+
+>>> device = "cuda" if torch.cuda.is_available() else "cpu"
+>>> converted_path = "/path/to/sam3.1-video-hf"
+
+>>> model = Sam3_1VideoModel.from_pretrained(converted_path).to(device)
+>>> image_processor = Sam3ImageProcessor(size={"height": 1008, "width": 1008})
+
+>>> image_url = "https://huggingface.co/datasets/hf-internal-testing/sam2-fixtures/resolve/main/truck.jpg"
+>>> image = Image.open(requests.get(image_url, stream=True).raw).convert("RGB")
+>>> pixel_values = image_processor(images=image, return_tensors="pt")["pixel_values"].to(device)
+
+>>> input_points = torch.tensor([[[[500.0, 375.0]]]], device=device)
+>>> input_labels = torch.tensor([[[1]]], device=device)
+
+>>> with torch.no_grad():
+...     outputs = model(
+...         pixel_values=pixel_values,
+...         input_points=input_points,
+...         input_labels=input_labels,
+...         run_propagation_head=True,
+...     )
+
+>>> print(outputs.interactive_pred_masks.shape)
+torch.Size([1, 1, 288, 288])
+>>> print(outputs.interactive_high_res_masks.shape)
+torch.Size([1, 1, 1008, 1008])
+>>> print(outputs.propagation_masks.shape)
+torch.Size([1, 1, 3, 288, 288])
+```
+
+### Box prompt
+
+```python
+>>> input_boxes = torch.tensor([[[150.0, 250.0, 900.0, 760.0]]], device=device)
+
+>>> with torch.no_grad():
+...     outputs = model(
+...         pixel_values=pixel_values,
+...         input_boxes=input_boxes,
+...         multimask_output=False,
+...     )
+
+>>> print(outputs.interactive_pred_masks.shape)
+torch.Size([1, 1, 288, 288])
+>>> print(outputs.interactive_iou_scores.shape)
+torch.Size([1, 1, 1])
+```
+
+## Sam3_1VideoConfig
+
+[[autodoc]] Sam3_1VideoConfig
+
+## Sam3_1VideoModel
+
+[[autodoc]] Sam3_1VideoModel
+    - forward
+
+## Sam3_1ViTModel
+
+[[autodoc]] Sam3_1ViTModel
+    - forward

plan.md

@@ -0,0 +1,124 @@
+# SAM 3.1 Support Plan
+
+Progress note written on 2026-03-28 after inspecting the current `sam3`, `sam3_video`, `sam3_tracker`, `sam3_tracker_video` Transformers implementations and comparing them against the upstream SAM 3.1 release in `/Users/nielsrogge/Documents/python_projecten/sam3`.
+
+## Policy note
+
+- If this work turns into a `transformers` PR, the repository's agent contribution policy requires coordination on the relevant issue first.
+- Violating that policy can lead to automatic banning, so the implementation work should stay in planning / local prototyping mode until coordination is clear.
+
+## Current findings
+
+- SAM 3.1 is not a drop-in replacement for the current SAM 3 video stack in Transformers.
+- Upstream changed the video path to Object Multiplex buckets, added a new multiplex predictor stack, and introduced a merged `sam3.1_multiplex.pt` checkpoint with custom remapping / non-strict loading behavior.
+- The image path is much closer to existing SAM 3 than the video path is.
+- Upstream `build_sam3_image_model()` still downloads `version="sam3"`, not `version="sam3.1"`.
+- The 3.1 detector used inside the new video stack is still `Sam3Image`-based; the biggest detector-side change is a tri-head neck that keeps the original detector neck as the main `convs` branch and adds `interactive_convs` and `propagation_convs`.
+- That strongly suggests `Sam3Model` is still the right class for SAM 3.1 image support, and that the main image-side work is likely checkpoint extraction / conversion rather than a new detector architecture.
+- Full SAM 3.1 video support is a new architecture from the Transformers point of view.
+- The current Transformers `sam3_video` and `sam3_tracker_video` code assume per-object tracking, not multiplex buckets.
+
+## Current repo structure implications
+
+- `src/transformers/models/sam3/` is only partially modular today. `modular_sam3.py` covers the image processor, but `modeling_sam3.py` is still hand-written.
+- `src/transformers/models/sam3_tracker/` is modular-based.
+- `src/transformers/models/sam3_tracker_video/` is modular-based.
+- `src/transformers/models/sam3_video/` is currently hand-written and does not have a modular file.
+- Because of that, the cleanest way to use modular for SAM 3.1 is probably to add new 3.1-specific video / tracker folders rather than first refactoring all existing SAM 3 code into modular form.
+
+## Recommended repository strategy
+
+- Keep SAM 3.1 image support inside the existing `src/transformers/models/sam3/` folder unless checkpoint inspection proves a real detector architecture mismatch.
+- Add a new video family under `src/transformers/models/sam3_1_video/`.
+- Only add a standalone `src/transformers/models/sam3_1_tracker_video/` folder if we decide that exposing the multiplex tracker directly is worth the extra API surface and maintenance cost.
+- If the standalone multiplex tracker is not needed publicly, keep tracker internals private to `sam3_1_video` and expose only the end-to-end video model / processor.
+- Use modular for new 3.1 tracker / video code where inheritance is helpful:
+- A new `sam3_1_tracker_video` modular file could inherit from `sam3_tracker_video` pieces and override config, state management, and feature plumbing.
+- A new `sam3_1_video` modular file can still import / inherit from current hand-written `sam3_video` classes even though `sam3_video` itself is not modular yet.
+- Avoid refactoring current `sam3_video` to modular as a prerequisite unless code duplication becomes severe; that is extra scope.
+
+## Work breakdown
+
+### 1. Checkpoint reconnaissance and mapping
+
+- Download or inspect the local `facebook/sam3.1` checkpoint and enumerate key prefixes and tensor shapes.
+- Separate detector weights, tri-head extra weights, and multiplex tracker weights.
+- Confirm whether the detector branch can be loaded into the current `Sam3Model` after conversion while ignoring video-only branches.
+- Draft exact mapping rules for:
+- `sam3.1` image extraction into `Sam3Model`
+- `sam3.1` video conversion into a new `Sam3_1VideoModel`
+
+### 2. Image support in existing `sam3`
+
+- Extend `src/transformers/models/sam3/convert_sam3_to_hf.py` so it can understand 3.1 checkpoint roots such as `detector.` or legacy `sam3_model.`.
+- Teach the converter to ignore or drop `interactive_convs` / `propagation_convs` when building an image-only detector checkpoint.
+- Validate whether `modeling_sam3.py` can stay unchanged.
+- If it cannot, make the smallest compatible detector-side change instead of creating a new 3.1 image model family.
+- Decide whether `facebook/sam3.1` should load directly into `Sam3Model`, or whether a separate converted detector repo id is cleaner for the first pass.
+- Add image integration tests for 3.1 loading and output parity checks against upstream image inference.
+
+### 3. New video family for SAM 3.1
+
+- Add `Sam3_1VideoConfig` with multiplex-specific fields such as:
+- `multiplex_count`
+- `max_num_objects`
+- dynamic bucket / planning settings
+- updated detection / reconditioning knobs
+- Implement a detector vision path that can expose detector, interactive, and propagation features from one backbone pass.
+- Decide whether to:
+- add a tri-head detector helper under the new video folder, or
+- minimally extend shared SAM 3 vision utilities in a reusable way
+- Implement multiplex inference session / state management:
+- bucketized object ids
+- add / remove / refine prompt flows
+- propagation lifecycle
+- cancellation / reset behavior
+- cached per-frame tri-head features
+- Implement the multiplex tracker core or wrapper.
+- Decide whether this lives in a standalone `sam3_1_tracker_video` public model or as private components inside `sam3_1_video`.
+- Implement the new video processor and post-processing logic.
+- Add a new conversion script for SAM 3.1 video checkpoints.
+
+### 4. Optional standalone multiplex tracker family
+
+- Only do this if there is a strong product / API reason to expose the multiplex tracker itself.
+- If yes, add `src/transformers/models/sam3_1_tracker_video/` with modular inheritance from `sam3_tracker_video`.
+- Keep the tracker API aligned with the new 3.1 video model so there is not a split maintenance story.
+- If no, skip this folder for the initial support pass and keep tracker internals private.
+
+### 5. Registrations, docs, and tests
+
+- Register any new model types in:
+- `src/transformers/models/auto/configuration_auto.py`
+- `src/transformers/models/auto/modeling_auto.py`
+- `src/transformers/models/auto/image_processing_auto.py`
+- `src/transformers/models/auto/processing_auto.py`
+- `src/transformers/models/auto/tokenization_auto.py`
+- `src/transformers/models/auto/video_processing_auto.py`
+- package `__init__.py` files
+- Update docs pages and `_toctree.yml`.
+- Add unit tests and integration tests mirroring current SAM 3 image / tracker / video coverage.
+- If a new folder uses modular, run the modular converter and then `make fix-repo`.
+- Finish with `make style` and targeted tests, then broader checks as needed.
+
+## Open design decisions
+
+- Should SAM 3.1 image support be wired to the existing `Sam3Model` only, or do we want a new config / model type for bookkeeping?
+- Do we want a public standalone `sam3_1_tracker_video` class, or is `sam3_1_video` enough for the first milestone?
+- Is it acceptable for the first implementation to support only the end-to-end multiplex video model and not every internal upstream class?
+- Should the user-facing processor / session API stay close to current `Sam3VideoProcessor`, or should it follow the upstream unified predictor flow more closely?
+- Do we want to refactor shared SAM 3 video utilities into reusable modular-friendly helpers before adding 3.1, or only after the first working version lands?
+
+## Suggested implementation order
+
+1. Inspect a real `sam3.1` checkpoint and write the concrete key-mapping rules.
+2. Land image support first by reusing `Sam3Model` if checkpoint extraction works.
+3. Prototype the new `sam3_1_video` config and conversion path.
+4. Implement multiplex state / session logic and tri-head detector feature reuse.
+5. Decide whether a standalone `sam3_1_tracker_video` public class is still needed.
+6. Add registrations, docs, and tests.
+7. Run `make fix-repo`, `make style`, and the relevant model tests.
+
+## Immediate next step
+
+- Inspect the actual local `sam3.1` checkpoint state dict and draft the exact conversion rules for image extraction and new video conversion before touching implementation files.