Feature Extraction Support for image models

[EFord36]

Feature request

I've been trying to use ORTModelForFeatureExtraction for a DinoV2 model, for which onyx export support was added in #2001 . However, when trying to run the model over input as for a transformers AutoModel, I get:

ValueError: Input pixel_values is required by model but not provided.

According to this comment feature extraction isn't supported for image models in general. Looking at ORTModelForFeatureExtraction.forward seems to confirm this. I couldn't find an issue tracking support for this for image models generally, so thought I would open one.

Motivation

I would like to be able to use an ONNX ORT model for feature extraction for DinoV2, but currently can't.

This seems to be general to image models.

Your contribution

I'm happy to work on a PR, but might need advise as to structure/style. E.g. would the maintainers prefer a new class ORTModelForImageFeatureExtraction, or for ORTModelForFeatureExtraction.forward to more dynamically set the model_inputs depending on what args are passed to the forward method?

[github-actions]

This issue has been marked as stale because it has been open for 30 days with no activity. This thread will be automatically closed in 5 days if no further activity occurs.

[EFord36]

Commenting to keep open: given that #2255 is now open to address this, and this is still a limitation I'm facing.

[IlyasMoutawwakil]

Hi, for now to unblock you could use the ORTModelForCustomTask class which should enable you to run any model with any inputs/outputs, the PR you mentioned will ofc make image feature extraction more accessible in the next release.