Skip to content

Write torch.profiler traces from a child process on multi-node trainers too - #366

Merged
Micah Yong (micahtyong) merged 1 commit into
mainfrom
micah/profile-fork-multinode
Nov 18, 2026
Merged

Micah Yong (micahtyong) merged 1 commit into
mainfrom
micah/profile-fork-multinode

Conversation

@micahtyong

Copy link
Copy Markdown
Collaborator

Summary

Multi-node Miles LoRA trainers now write the profiled step's trace from a child process, the same way single-node trainers do since #359. Before this change they wrote it in place, and every client waited for the write.

#359 kept the in-place path for multi-node trainers. The concern was that forking a process that uses RDMA can corrupt its registered memory. A 2-node test shows that doesn't happen here. NCCL calls ibv_fork_init, so the RDMA library marks each registered buffer MADV_DONTFORK, and the sandbox Modal runs containers in honors that flag. The forked child never maps those buffers, so the parent keeps its own pages. Newer sandbox releases also copy pinned pages at fork, as Linux 5.9+ does. Either way the parent's buffers stay intact.

This removes the fork flag, the in-place branch and its test.

Testing

The test ran a 2-node trainer with 8 H200s per node: Qwen3.5-2B LoRA with data-parallel 16 over RDMA. Inter-node NCCL channels ran over NET/IB/GDRDMA on mlx5 RoCE. The client was the stock Tinker SDK running an SFT loop with create_lora_training_client(seed=1234), fixed data and 32 × 1024 tokens per step. Commit d23724c8 has the same code as this PR.

  • Profiled run: SPINDLE_TORCH_PROFILE_STEP=2 and SPINDLE_TORCH_PROFILE_RANKS=all, 12 steps (W&B). After that, another 12-step run on the same trainer (W&B).
  • Control: two unprofiled runs on a separate 2-node trainer (W&B, W&B).
  • All 16 ranks forked a writer. Each wrote its trace in 5.0 s and exited 0. All 32 files, including those from the second node, were on the checkpoint volume afterwards.
  • In 10 parent/child pairs captured at fork time, the child was missing about 826 of the parent's mappings (194 to 322 MiB). Those are the MADV_DONTFORK ranges. The sandbox's smaps doesn't print the dc flag, so the check compared /proc/<pid>/maps instead.
  • Loss and grad norm matched the control on every step after the fork. Loss was within 1.3e-5 relative, and the two controls differed by up to 2.6e-5. Grad norm was within 0.5%, and the controls differed by up to 0.35%. The post-fork run stayed within the same range.
  • There were no NCCL warnings, timeouts or hangs.
step 2 (profiled) step 3 step 4 steps 5–11
fwd_bwd + optim, profiled 2.9 s 3.4 s 2.5 s 1.9–2.6 s
fwd_bwd + optim, control 1.7–2.0 s 1.8 s 1.7–1.8 s 1.7–2.7 s

Not run: forking on an EFA cluster. There, aws-ofi-nccl sets FI_EFA_FORK_SAFE=1, and libfabric then calls ibv_fork_init or fails provider init.

…rs too

Multi-node trainers fork the trace writer like single-node ones. A 2-node
RDMA trainer showed the registered memory is kept out of the child, all 16
ranks' traces landed, and training matched an unprofiled control.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@micahtyong
Micah Yong (micahtyong) merged commit 7c98b4e into main Nov 18, 2026
3 checks passed
@micahtyong
Micah Yong (micahtyong) deleted the micah/profile-fork-multinode branch November 18, 2026 01:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant