Skip to content

Warn when resuming from an older checkpoint - #50766

Open
bezabyte wants to merge 1 commit into
huggingface:mainfrom
bezabyte:codex/warn-older-checkpoint-resume
Open

bezabyte wants to merge 1 commit into
huggingface:mainfrom
bezabyte:codex/warn-older-checkpoint-resume

Conversation

@bezabyte

@bezabyte bezabyte commented Nov 14, 2026 •

Copy link
Copy Markdown

🤗 Transformers CI

Fixes #50677

When Trainer resumes from an older checkpoint in the same output_dir while newer checkpoints exist, log a warning explaining that later saves can overwrite those checkpoints and that resume_from_checkpoint=True may select a newer checkpoint after an interruption.

This also documents using a new output_dir for rollback and adds a regression test.

Validation:

  • Direct rollback-warning smoke test passed.
  • python -m compileall -q src/transformers/trainer.py tests/trainer/test_trainer_checkpointing.py passed.
  • git diff --check passed.
  • Full Trainer test could not run locally because accelerate is not installed; pytest collection also hits the environment's missing xdist hook.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution 🤗!

CI Security Gate — waiting for maintainer approval

CI is approved automatically only for authors who already have a contribution merged in this repository. A maintainer will review this PR and approve CI to run on it.

@bezabyte

Copy link
Copy Markdown
Author
before-resume-checkpoint after-resume-checkpoint

Before / after comparison:

  • Before: resuming from checkpoint-2 while newer checkpoints existed emitted no warning.
  • After: Trainer warns about the newer checkpoints and recommends a new output_dir for rollback.
    These screenshots come from the rollback-warning smoke test.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Trainer: after resume_from_checkpoint=<older>, a crash while re-saving checkpoint-N can silently mix two runs

2 participants