Skip to content

synmon: resume swe-gym from checkpoints after trainer preemption - #303

Draft
Micah Yong (micahtyong) wants to merge 1 commit into
mainfrom
micah/engine-redelivery-fence
Draft

Micah Yong (micahtyong) wants to merge 1 commit into
mainfrom
micah/engine-redelivery-fence

Conversation

@micahtyong

Copy link
Copy Markdown
Collaborator

Summary

The 2026-11-16 nightly swe-gym run failed after two preemptions:

  • The CPU container running the swe-gym client was preempted at step 4 of 5, and Modal restarted the run from scratch.
  • On the second attempt, the 8xH200 trainer was preempted. Both LoRA models lost their GPU state, and the next forward_backward returned HTTP 410 model_lost.

Losing models on a GPU preemption is expected. The swe-gym example already saves state after every step and can continue with --resume. This PR makes the nightly harness use that path:

  • monitor is now nonpreemptible, so a client run no longer restarts from scratch.
  • When swe-gym exits with model_lost, the harness reruns the stock example once with --resume, using the remaining time budget. Each client continues from its last checkpoint. Other failures still fail the night.
  • A resumed night shows its runtime with "resumed from checkpoints, not compared", plus the error that triggered the resume. It does not become a timing baseline or fail the cron run.
  • Client stderr still streams to the logs. Failure messages now show the client's final exception line instead of the command line.

E2E

  • Replayed the preemption with a stand-in client that prints the real 410 model_lost error, exits 1, and then finishes its remaining steps on --resume. The harness resumed once, counted 5 steps per client, and reported "resumed from checkpoints, not compared".
  • A stand-in client that fails with a different exception is not resumed. The failure is reported as exit 1: helpers.environments.InfrastructureError: Nonfinite training metrics.
  • Not run: a live resume against the prod trainer after a real preemption.

🤖 Generated with Claude Code

…tor nonpreemptible

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant