Repository navigation
Write torch.profiler traces from a child process on multi-node trainers too - #366
Merged
Merged
Conversation
…rs too Multi-node trainers fork the trace writer like single-node ones. A 2-node RDMA trainer showed the registered memory is kept out of the child, all 16 ranks' traces landed, and training matched an unprofiled control. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Micah Yong (micahtyong)
requested a review
from Kailash Ranganathan (kailash109)
as a code owner
November 18, 2026 01:12
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Multi-node Miles LoRA trainers now write the profiled step's trace from a child process, the same way single-node trainers do since #359. Before this change they wrote it in place, and every client waited for the write.
#359 kept the in-place path for multi-node trainers. The concern was that forking a process that uses RDMA can corrupt its registered memory. A 2-node test shows that doesn't happen here. NCCL calls
ibv_fork_init, so the RDMA library marks each registered bufferMADV_DONTFORK, and the sandbox Modal runs containers in honors that flag. The forked child never maps those buffers, so the parent keeps its own pages. Newer sandbox releases also copy pinned pages at fork, as Linux 5.9+ does. Either way the parent's buffers stay intact.This removes the
forkflag, the in-place branch and its test.Testing
The test ran a 2-node trainer with 8 H200s per node: Qwen3.5-2B LoRA with data-parallel 16 over RDMA. Inter-node NCCL channels ran over
NET/IB/GDRDMAon mlx5 RoCE. The client was the stock Tinker SDK running an SFT loop withcreate_lora_training_client(seed=1234), fixed data and 32 × 1024 tokens per step. Commit d23724c8 has the same code as this PR.SPINDLE_TORCH_PROFILE_STEP=2andSPINDLE_TORCH_PROFILE_RANKS=all, 12 steps (W&B). After that, another 12-step run on the same trainer (W&B).MADV_DONTFORKranges. The sandbox'ssmapsdoesn't print thedcflag, so the check compared/proc/<pid>/mapsinstead.Not run: forking on an EFA cluster. There, aws-ofi-nccl sets
FI_EFA_FORK_SAFE=1, and libfabric then callsibv_fork_initor fails provider init.