Skip to content
Merged
Show file tree
Hide file tree
Changes from 34 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
54c815f
fixes for transformers 4.54 and 5.55
IlyasMoutawwakil Aug 9, 2025
2fbc83e
Merge branch 'main' into transformers-4.55
IlyasMoutawwakil Sep 8, 2025
c2de155
test 4.54
IlyasMoutawwakil Sep 8, 2025
075247d
don't use bigcode branch
IlyasMoutawwakil Sep 8, 2025
085d871
fix
IlyasMoutawwakil Sep 8, 2025
df15230
fix
IlyasMoutawwakil Sep 8, 2025
f06a169
Merge branch 'main' into transformers-4.55
IlyasMoutawwakil Sep 8, 2025
6ccc95f
fix
IlyasMoutawwakil Sep 8, 2025
5ac0170
fix
IlyasMoutawwakil Sep 8, 2025
fd2a30d
fix
IlyasMoutawwakil Sep 8, 2025
983ce8d
set pkv to None when it's empty
IlyasMoutawwakil Sep 8, 2025
389be21
same in decoder
IlyasMoutawwakil Sep 8, 2025
c7b64e0
4.56 bcz why not
IlyasMoutawwakil Sep 8, 2025
e6281c5
fix
IlyasMoutawwakil Sep 8, 2025
9abe096
test
IlyasMoutawwakil Sep 9, 2025
b68ff86
fix
IlyasMoutawwakil Sep 9, 2025
e89b44f
add dtype
IlyasMoutawwakil Sep 9, 2025
3a1f0ac
add test separation for seq2seq
IlyasMoutawwakil Sep 9, 2025
8a0239e
fix
IlyasMoutawwakil Sep 9, 2025
d674231
fix
IlyasMoutawwakil Sep 9, 2025
cedb778
style
IlyasMoutawwakil Sep 9, 2025
07ac109
limit to transformers 4.55
IlyasMoutawwakil Sep 9, 2025
5080be8
fix
IlyasMoutawwakil Sep 9, 2025
52ac0e0
fix
IlyasMoutawwakil Sep 9, 2025
747d7cd
add gpt_oss
IlyasMoutawwakil Sep 9, 2025
f378ca7
fix sam
IlyasMoutawwakil Sep 10, 2025
5ac17c0
fix gpt-oss
IlyasMoutawwakil Sep 10, 2025
0a5ffba
fix moonshine and comment imagegpt
IlyasMoutawwakil Sep 10, 2025
23b0700
not sure why it's failing on ci
IlyasMoutawwakil Sep 10, 2025
acfda06
test
IlyasMoutawwakil Sep 10, 2025
b6f80bf
remove
IlyasMoutawwakil Sep 10, 2025
aa619c3
simpler old modeling
IlyasMoutawwakil Sep 11, 2025
4bc8bd9
fix
IlyasMoutawwakil Sep 11, 2025
e791f8b
remove 4.54
IlyasMoutawwakil Sep 11, 2025
2f8ad72
Update optimum/exporters/onnx/_traceable_decorator.py
IlyasMoutawwakil Sep 11, 2025
f06d999
nitpicks and naming
IlyasMoutawwakil Sep 11, 2025
c1f0c14
licences
IlyasMoutawwakil Sep 12, 2025
3775b6e
comments
IlyasMoutawwakil Sep 12, 2025
6788154
warnings for old modeling
IlyasMoutawwakil Sep 12, 2025
96040e7
comment
IlyasMoutawwakil Sep 12, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 7 additions & 5 deletions .github/workflows/test_exporters_common.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,18 @@ concurrency:
cancel-in-progress: true

env:
UV_SYSTEM_PYTHON: true
UV_TORCH_BACKEND: auto
TRANSFORMERS_IS_CI: true
HF_HUB_READ_TOKEN: ${{ secrets.HF_HUB_READ_TOKEN }}

jobs:
build:
strategy:
fail-fast: false
matrix:
runs-on: [ubuntu-22.04]
python-version: [3.9]
runs-on: [ubuntu-22.04]

runs-on: ${{ matrix.runs-on }}

Expand All @@ -34,10 +37,9 @@ jobs:

- name: Install dependencies
run: |
pip install --upgrade pip
pip install --no-cache-dir torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install .[tests,onnxruntime]
pip install --upgrade pip uv
uv pip install .[tests,onnxruntime]

- name: Test with pytest
run: |
pytest tests/exporters/common -vvvv --durations=0 -n auto
pytest tests/exporters/common -vvvv -n auto
10 changes: 6 additions & 4 deletions .github/workflows/test_exporters_onnx.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,10 @@ concurrency:
cancel-in-progress: true

env:
UV_SYSTEM_PYTHON: true
UV_TORCH_BACKEND: auto
TRANSFORMERS_IS_CI: true
HF_HUB_READ_TOKEN: ${{ secrets.HF_HUB_READ_TOKEN }}

jobs:
build:
Expand All @@ -34,10 +37,9 @@ jobs:

- name: Install dependencies
run: |
pip install --upgrade pip
pip install --no-cache-dir torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install .[tests,onnxruntime] diffusers
pip install --upgrade pip uv
uv pip install .[tests,onnxruntime] diffusers

- name: Test with pytest
run: |
pytest tests/exporters/onnx/test_export.py -vvvv --durations=0 -n auto
pytest tests/exporters/onnx/test_export.py -vvvv -n auto
14 changes: 8 additions & 6 deletions .github/workflows/test_exporters_onnx_cli.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,17 +11,20 @@ concurrency:
cancel-in-progress: true

env:
UV_SYSTEM_PYTHON: true
UV_TORCH_BACKEND: auto
TRANSFORMERS_IS_CI: true
HF_HUB_READ_TOKEN: ${{ secrets.HF_HUB_READ_TOKEN }}

jobs:
build:
strategy:
fail-fast: false
matrix:
python-version: [3.9]
os: [ubuntu-22.04]
runs-on: [ubuntu-22.04]

runs-on: ${{ matrix.os }}
runs-on: ${{ matrix.runs-on }}

steps:
- name: Checkout repository
Expand All @@ -34,10 +37,9 @@ jobs:

- name: Install dependencies
run: |
pip install --upgrade pip
pip install --no-cache-dir torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install .[tests,onnxruntime] diffusers
pip install --upgrade pip uv
uv pip install .[tests,onnxruntime] diffusers

- name: Test with pytest
run: |
pytest tests/exporters/onnx/test_export_cli.py -vvvv --durations=0 -n auto
pytest tests/exporters/onnx/test_export_cli.py -vvvv -n auto
37 changes: 23 additions & 14 deletions .github/workflows/test_onnxruntime.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,10 @@ concurrency:
cancel-in-progress: true

env:
UV_SYSTEM_PYTHON: true
UV_TORCH_BACKEND: auto
TRANSFORMERS_IS_CI: true
HF_HUB_READ_TOKEN: ${{ secrets.HF_HUB_READ_TOKEN }}

jobs:
build:
Expand All @@ -24,22 +27,19 @@ jobs:
transformers_version: [latest, 4.36.*, 4.45.*]
test_file:
[
test_timm.py,
test_decoder.py,
test_modeling.py,
test_diffusion.py,
test_modeling.py,
test_optimization.py,
test_quantization.py,
test_seq2seq.py,
test_timm.py,
test_utils.py,
]

runs-on: ${{ matrix.runs-on }}

steps:
- name: Free Disk Space (Ubuntu)
if: matrix.test_file == 'test_modeling.py'
uses: jlumbroso/free-disk-space@main

- name: Checkout code
uses: actions/checkout@v5

Expand All @@ -50,20 +50,29 @@ jobs:

- name: Install dependencies
run: |
pip install --upgrade pip
pip install --no-cache-dir torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
pip install .[tests,onnxruntime] diffusers
pip install --upgrade pip uv
uv pip install .[tests,onnxruntime]

- name: Install diffusers for transformers ${{ matrix.transformers-version }}
run: |
if [ "${{ matrix.transformers_version }}" == '4.36.*' ]; then
uv pip install "diffusers<0.32.0"
elif [ "${{ matrix.transformers_version }}" == '4.45.*' ]; then
uv pip install "diffusers<0.33.0"
else
uv pip install diffusers
fi

- name: Install transformers ${{ matrix.transformers-version }}
run: |
if [ "${{ matrix.transformers_version }}" == '4.36.*' ]; then
pip install "transformers==4.36.*" "diffusers<0.32.0" "pytest<8.0.0"
uv pip install "transformers==4.36.*" "pytest<8.0.0"
elif [ "${{ matrix.transformers_version }}" == '4.45.*' ]; then
pip install "transformers==4.45.*" "diffusers<0.33.0"
uv pip install "transformers==4.45.*"
elif [ "${{ matrix.transformers_version }}" != 'latest' ]; then
uv pip install "transformers==${{ matrix.transformers_version }}"
fi

- name: Test with pytest
run: |
pytest tests/onnxruntime/${{ matrix.test_file }} --durations=0 -vvvv -n auto
env:
HF_HUB_READ_TOKEN: ${{ secrets.HF_HUB_READ_TOKEN }}
pytest tests/onnxruntime/${{ matrix.test_file }} -vvvv -n auto
153 changes: 153 additions & 0 deletions optimum/exporters/onnx/_traceable_decorator.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
import warnings
from collections import defaultdict
from functools import wraps

from transformers.utils.generic import _CAN_RECORD_REGISTRY, OutputRecorder, logger


def check_model_inputs_patched(func):
@wraps(func)
def wrapper(self, *args, **kwargs):
use_cache = (
kwargs["use_cache"] if kwargs.get("use_cache") is not None else getattr(self.config, "use_cache", None)
)
if use_cache is not None:
if getattr(self, "gradient_checkpointing", False) and self.training and use_cache:
Comment thread
IlyasMoutawwakil marked this conversation as resolved.
Resolved
logger.warning_once(
"`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`."
)
use_cache = False

# Prevent passing use_cache twice
if "use_cache" in func.__code__.co_varnames:
use_cache_idx = func.__code__.co_varnames.index("use_cache") - 1 # minus 1 for 'self'
if len(args) > use_cache_idx:
args = list(args)
args[use_cache_idx] = use_cache
args = tuple(args)
else:
kwargs["use_cache"] = use_cache

return_dict = kwargs.pop("return_dict", None)
if return_dict is None:
return_dict = getattr(self.config, "return_dict", True)

all_args = kwargs.copy()
if "kwargs" in all_args:
for k, v in all_args["kwargs"].items():
all_args[k] = v

capture_flags = _CAN_RECORD_REGISTRY.get(str(self.__class__), {}) # there is a weak ref for executorch
if capture_flags is None:
capture_flags = {}
Comment thread
IlyasMoutawwakil marked this conversation as resolved.
Resolved

recordable_keys = {
f"output_{k}": all_args.get(
f"output_{k}",
getattr(
self.config,
f"output_{k}",
all_args.get("output_attentions", getattr(self.config, "output_attentions", False)),
),
)
for k in capture_flags
}

# We let cross attentions to be saved separately because some models add `cross-attn` layer
# when certain condtions are met. Let's output cross attention if attentions are requested (for BC)
if "output_attentions" in recordable_keys:
recordable_keys["output_cross_attentions"] = recordable_keys["output_attentions"]

collected_outputs = defaultdict(tuple)
monkey_patched_layers = []

# Check attention implementation is properly set for capturing attention outputs
if recordable_keys.get("output_attentions", False):
supported_attn = ["eager", "eager_paged", "flex_attention"]
config_attn = getattr(self.config, "_attn_implementation", None)
sub_configs = [getattr(self.config, key, None) for key in self.config.sub_configs]
sub_configs_attn = [
getattr(config, "_attn_implementation", None) for config in sub_configs if config is not None
]
if config_attn not in supported_attn or any(attn not in supported_attn for attn in sub_configs_attn):
warnings.warn(
f"`output_attentions=True` is not supported with `attn_implementation` other than {supported_attn}. "
"Please use `model.set_attn_implementation('eager')` to enable capturing attention outputs.",
UserWarning,
stacklevel=2,
)

def make_capture_wrapper(module, orig_forward, key, index):
@wraps(orig_forward)
def wrapped_forward(*args, **kwargs):
if key == "hidden_states" and len(collected_outputs[key]) == 0:
collected_outputs[key] += (args[0],)
output = orig_forward(*args, **kwargs)
if not isinstance(output, tuple):
collected_outputs[key] += (output,)
elif output[index] is not None:
if key not in collected_outputs:
collected_outputs[key] = (output[index],)
else:
collected_outputs[key] += (output[index],)
return output

return wrapped_forward

if any(recordable_keys.values()):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure to understand why we need all of this, if you could add some comment to explain that would be awesome 🙏

@IlyasMoutawwakil IlyasMoutawwakil Sep 12, 2025 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just patched/fixed the function to accept use_cache as a positional argument and not fail with some models 😭
it's a new transformers decorator that was added around all forward passes to handle a lot of the use_cache, rtutn_tuple, output_attentions, .. arguments in one central place.
it's very hacky and breaks a couple things unfortunately but we can't remove it unfortunately as it now handles a big chunk of the "what should the model output and in what format" logic.

capture_tasks = []
for key, layer_specs in capture_flags.items():
if not recordable_keys.get(f"output_{key}", False):
continue
if not isinstance(layer_specs, list):
layer_specs = [layer_specs]
for specs in layer_specs:
if not isinstance(specs, OutputRecorder):
index = 0 if "hidden_states" in key else 1
class_name = None if not isinstance(specs, str) else specs
target_class = specs if not isinstance(specs, str) else None
specs = OutputRecorder(target_class=target_class, index=index, class_name=class_name)
capture_tasks.append((key, specs))

for name, module in self.named_modules():
for key, specs in capture_tasks:
# The second check is for multimodals where only backbone layer suffix is available
if (specs.target_class is not None and isinstance(module, specs.target_class)) or (
specs.class_name is not None and name.endswith(specs.class_name)
):
if specs.layer_name is not None and specs.layer_name not in name:
continue
# Monkey patch forward
original_forward = module.forward
module.forward = make_capture_wrapper(module, original_forward, key, specs.index)
monkey_patched_layers.append((module, original_forward))

outputs = func(self, *args, **kwargs)
# Restore original forward methods
for module, original_forward in monkey_patched_layers:
module.forward = original_forward

# Inject collected outputs into model output
for key in collected_outputs:
if key == "hidden_states":
if hasattr(outputs, "vision_hidden_states"):
collected_outputs[key] = collected_outputs[key][:-1]
collected_outputs[key] += (outputs.vision_hidden_states,)
elif hasattr(outputs, "last_hidden_state"):
collected_outputs[key] = collected_outputs[key][:-1]
collected_outputs[key] += (outputs.last_hidden_state,)

outputs[key] = collected_outputs[key]
elif key == "attentions":
if isinstance(capture_flags[key], list) and len(capture_flags[key]) == 2:
outputs[key] = collected_outputs[key][0::2]
outputs["cross_" + key] = collected_outputs[key][1::2]
else:
outputs[key] = collected_outputs[key]
else:
outputs[key] = collected_outputs[key]
if return_dict is False:
outputs = outputs.to_tuple()
return outputs

return wrapper
53 changes: 53 additions & 0 deletions optimum/exporters/onnx/input_generators.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
from __future__ import annotations
Comment thread
IlyasMoutawwakil marked this conversation as resolved.
Resolved

from optimum.utils import DummyPastKeyValuesGenerator, NormalizedTextConfig, is_transformers_version


class GPTBigCodeDummyPastKeyValuesGenerator(DummyPastKeyValuesGenerator):
def __init__(self, task: str, normalized_config: NormalizedTextConfig, **kwargs):
super().__init__(task=task, normalized_config=normalized_config, **kwargs)
self.multi_query = normalized_config.multi_query

def generate(self, input_name: str, framework: str = "pt", int_dtype: str = "int64", float_dtype: str = "fp32"):
if is_transformers_version("<", "4.54"):
if self.multi_query:
shape = (
self.batch_size,
self.sequence_length,
self.hidden_size // self.num_attention_heads * 2,
)
else:
shape = (
self.batch_size,
self.num_attention_heads,
self.sequence_length,
self.hidden_size // self.num_attention_heads * 2,
)
pkv = [
self.random_float_tensor(shape, framework=framework, dtype=float_dtype) for _ in range(self.num_layers)
]

else:
if self.multi_query:
shape = (
self.batch_size,
1,
self.sequence_length,
self.hidden_size // self.num_attention_heads,
)
else:
shape = (
self.batch_size,
self.num_attention_heads,
self.sequence_length,
self.hidden_size // self.num_attention_heads,
)
pkv = [
(
self.random_float_tensor(shape, framework=framework, dtype=float_dtype),
self.random_float_tensor(shape, framework=framework, dtype=float_dtype),
)
for _ in range(self.num_layers)
]

return pkv
Loading