Skip to content

Harden Beam 256k LoRA training, resume, and rollout defaults - #766

Merged
zhouhelena1 merged 11 commits into
mainfrom
codex/beam-lora-guards
Oct 28, 2026
Merged

zhouhelena1 merged 11 commits into
mainfrom
codex/beam-lora-guards

Conversation

@zhouhelena1

@zhouhelena1 zhouhelena1 commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Fix Beam 256k LoRA training and resume while retaining upstream’s batch size of 256.

  • Save complete native adapter/optimizer/scheduler checkpoints each update; restore the rollout cursor and prefetched-data position on retry. Reject incomplete checkpoints and unsafe resume overrides.
  • Validate CP/sequence-parallel padding; use Beam’s upstream attention backend and tool/reasoning parsers.
  • Remove the eight-request rollout cap, preserve required recipe defaults under overrides, and honor custom rewards.
  • Reserve CPU/RAM and retain the image’s RDMA stack since peak RAM is 248 GiB
  • Derive LR decay from the training horizon, enable configured evaluations, allow smaller refill batches, and correct debug GPU allocation/dashboard pricing.

Validation: full three-step run, logprob difference 0.00725–0.00935, KL 0.000174–0.000217

@github-actions

Copy link
Copy Markdown

@zhouhelena1 zhouhelena1 changed the title Guard Beam CP padding and disable unsafe LoRA retries Guard Beam CP padding and restore native LoRA checkpoints Oct 28, 2026
@zhouhelena1
zhouhelena1 deployed to modal-preview October 28, 2026 02:45 — with GitHub Actions Active

This branch was successfully deployed

1 active deployment
modal-preview — 05461858 Deployed Oct 28, 2026 by zhouhelena1 via preview #7355
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants