Skip to content

Add image classifier donut & update loss calculation for all swins #37224

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
zucchini-nlp merged 17 commits into from
Apr 10, 2025
Merged

Add image classifier donut & update loss calculation for all swins #37224

zucchini-nlp merged 17 commits into from
Apr 10, 2025

Conversation

@eljandoubi
Copy link
Contributor

@eljandoubi eljandoubi commented Apr 2, 2025 •
edited
Loading

What does this PR do?

  • Add classifier head to Donut
  • Add image classifier loss to LOSS_MAPPING
  • Update classifier loss for all swin models

Models:

@github-actions
Copy link
Contributor

github-actions bot commented Apr 2, 2025

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the Ready for review button (at the bottom of the PR page). This will assign reviewers and trigger CI.

@github-actions github-actions bot marked this pull request as draft April 2, 2025 22:47
@eljandoubi eljandoubi marked this pull request as ready for review April 3, 2025 06:29
@github-actions github-actions bot requested review from ydshieh and zucchini-nlp April 3, 2025 06:29
zucchini-nlp
zucchini-nlp reviewed Apr 3, 2025
View reviewed changes
src/transformers/models/donut/modeling_donut_swin.py
_EXPECTED_OUTPUT_SHAPE = [1, 49, 768]

# Image classification docstring
_IMAGE_CLASS_CHECKPOINT = "eljandoubi/donut-base-encoder"
Copy link
Member

@zucchini-nlp zucchini-nlp Apr 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this a checkpoint trained for classification or base model ckpt?

Copy link
Contributor Author

@eljandoubi eljandoubi Apr 3, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Traind base model with randomly initialized head. It is meant for further fine tuning.

@eljandoubi eljandoubi changed the title Add image classifier donut Add image classifier donut & update loss calculation for all swins Apr 3, 2025
zucchini-nlp
zucchini-nlp approved these changes Apr 3, 2025
View reviewed changes
Copy link
Member

@zucchini-nlp zucchini-nlp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, looks good to me! Requesting review from core maintainer

@eljandoubi
Copy link
Contributor Author

eljandoubi commented Apr 8, 2025 •
edited
Loading

@zucchini-nlp @ArthurZucker @ydshieh any feedback?

@zucchini-nlp
Copy link
Member

zucchini-nlp commented Apr 10, 2025

sorry, let's merge it. Doesn't touch much core code, should be good to go

@zucchini-nlp zucchini-nlp merged commit 7ecc5b8 into huggingface:main Apr 10, 2025
18 checks passed
cyr0930 pushed a commit to cyr0930/transformers that referenced this pull request Apr 18, 2025
@eljandoubi
Add image classifier donut & update loss calculation for all swins (h…
16ddf8e 
…uggingface#37224)

* add classifier head to donut

* add to transformers __init__

* add to auto model

* fix typo

* add loss for image classification

* add checkpoint

* remove no needed import

* reoder import

* format

* consistency

* add test of classifier

* add doc

* try ignore

* update loss for all swin models
zucchini-nlp pushed a commit to zucchini-nlp/transformers that referenced this pull request May 14, 2025
@eljandoubi @zucchini-nlp
Add image classifier donut & update loss calculation for all swins (h…
d1225ec 
…uggingface#37224)

* add classifier head to donut

* add to transformers __init__

* add to auto model

* fix typo

* add loss for image classification

* add checkpoint

* remove no needed import

* reoder import

* format

* consistency

* add test of classifier

* add doc

* try ignore

* update loss for all swin models
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zucchini-nlp zucchini-nlp zucchini-nlp approved these changes

@ydshieh ydshieh Awaiting requested review from ydshieh

@ArthurZucker ArthurZucker Awaiting requested review from ArthurZucker

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@eljandoubi @zucchini-nlp