Skip to content

Keep Miles LoRA trainers serving while a profiled step's trace is written - #359

Merged
Micah Yong (micahtyong) merged 3 commits into
mainfrom
micah/trainer-profile-async-export
Nov 18, 2026
Merged

Micah Yong (micahtyong) merged 3 commits into
mainfrom
micah/trainer-profile-async-export

Conversation

@micahtyong

@micahtyong Micah Yong (micahtyong) commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

After a profiled step, a Miles LoRA trainer stopped serving every client while it wrote the trace: 34 s on a 2B model, minutes on large ones.

Now the trainer stops the profiler on its own thread, then a forked child process writes the trace and table, so the trainer takes the next request right away. A background thread doesn't work, because building the table holds Python's global lock. Multi-node trainers still write in place, since forking a process that uses RDMA hasn't been verified. Tensor shapes and memory are no longer recorded by default (SPINDLE_TORCH_PROFILE_SHAPES=1 / SPINDLE_TORCH_PROFILE_MEMORY=1 turn them on).

Testing

Qwen3.5-2B LoRA, one H100, two stock Tinker SDK clients on one trainer, profiling step 3:

Trace per rank (gzip) Trainer unavailable Other client's next forward_backward W&B
main d00dcdb 8.7 MB 34.2 s 36.8 s A, B
This PR (8a6ea2c) 6.6 MB 1.1 s 4.3 s A, B

A normal forward_backward with two clients sharing the trainer takes 1.3–2.9 s. With this PR the files land 18 s after the stop. Writing from a background thread instead made the trainer's next forward_backward take 25 s (A).

Not run: multi-GPU and multi-node trainers, all ranks, large models, and a shutdown while files are still being written.

… serving

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@micahtyong
Micah Yong (micahtyong) merged commit ad19b23 into main Nov 18, 2026
3 checks passed
@micahtyong
Micah Yong (micahtyong) deleted the micah/trainer-profile-async-export branch November 18, 2026 00:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant