Repository navigation
[AMD] Fall back to mcore async checkpointing when NVRx is unavailable - #2816
Draft
michaelzhang-ai wants to merge 1 commit into
Draft
michaelzhang-ai wants to merge 1 commit into
michaelzhang-ai wants to merge 1 commit into
Conversation
Megatron defaults --async-strategy to "nvrx", which hard-requires nvidia-resiliency-ext. docker/Dockerfile.rocm strips that CUDA-only package from requirements.txt, so on ROCm every --async-save run reached the first checkpoint write and died with: ModuleNotFoundError: A compatible `nvidia-resiliency-ext` installation is required for `async_strategy="nvrx"` Probe for the NVRx async-checkpoint API in set_default_megatron_args and drop back to Megatron's in-tree "mcore" saver when it is missing. CUDA images ship the package, so they keep using nvrx unchanged.
2 of 5 tasks
michaelzhang-ai
marked this pull request as ready for review
August 30, 2026 05:49
michaelzhang-ai
requested review from
Shi-Dong,
Zhichenzzz,
fzyzcjy,
maocheng23,
yueming-yuan and
yushengsu-thu
as code owners
August 30, 2026 05:49
michaelzhang-ai
marked this pull request as draft
August 30, 2026 19:35
This was referenced Oct 25, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Warning
Do not merge as-is. Hardware verification showed this change is necessary but not sufficient: it removes the
ModuleNotFoundErrorbut the run then hangs for 90 minutes on a second, independent ROCm defect. Merging only this would trade a fast, legible failure for a silent hang, which is worse. See Verification — the companion Megatron change is identified precisely and this PR should land with it, or be re-scoped.Summary
--async-saveis broken on every ROCm image. Megatron defaults--async-strategyto"nvrx", which hard-requiresnvidia-resiliency-ext;docker/Dockerfile.rocmdeliberately strips that CUDA-only package out ofrequirements.txt:https://github.com/radixark/miles/blob/main/docker/Dockerfile.rocm#L167
So the run parses fine, trains fine, and then dies at the first checkpoint write:
set_default_megatron_argsnow probes for the NVRx async-checkpoint API and falls back to Megatron's in-tree"mcore"saver when it is missing. CUDA images ship the package, so they keep usingnvrxand are untouched.How this surfaced
tests/e2e/ckpt/test_qwen3_4B_ckpt.pyin the MI350 nightly, which runsexecute("save")→execute("load")→execute("async_save")→execute("load"). The first nightlies to run on a post-bump image all failed on the third step:test_qwen3_4B_ckpt.pymiles-rocm720-mi35x-20260824miles-rocm720-mi35x-20260824miles-rocm720-mi35x-20260828miles-rocm720-mi35x-20260829Failing run: sgl-project/sglang actions/runs/33265909152, job
nightly-stage-c-8-gpu-mi350.The 8/25–8/27 nightlies reused the stale
-20260824tag because the ROCm image builds for those days had failed, which is why a change that landed on 8/24 only turned the nightly red on 8/28.Why the bump changed behaviour
async_strategydid not exist before the Megatron bump in #2673. Counting the Megatron argument dumps in each nightly's 8-GPU log makes the transition explicit — same test set, same 26 training invocations, exactly one of which asks for async save:20260824(8/27, passed)20260829(8/29, failed)async_save ... Falseasync_save ... Trueasync_strategy ... mcoreasync_strategy ... nvrxOn the old image the option is absent entirely, so async save always used the in-tree writer. On the new image the 24 sync-save runs get
mcore(Megatron's own validation rewritesnvrx→mcorewheneverasync_saveis off), and the single async-save run keeps thenvrxdefault and hits the missing module.Verification
Run on real MI350 hardware against the nightly's own image (
rocm/sgl-dev:miles-rocm720-mi35x-20260829, torch2.9.1+rocm7.2.0, hip7.2.26015), by patching the in-image/root/milesfromrefs/pull/2816/headand measuring before and after in the same container: sgl-project/sglang actions/runs/33305121208.What works. The stock image reproduces the nightly's exception exactly, and the patch clears it:
nvidia_resiliency_ext presentresolved async_strategynvrxmcorebackend importModuleNotFoundError: A compatible nvidia-resiliency-ext installation is required...loaded mcoreAll five new unit tests also ran in that container against the real Megatron rather than being skipped by their
importorskip. In the end-to-end run, all four Megatron invocations reportedasync_strategy ... mcore, including the singleasync_save=Trueone — before the patch that one wasnvrx.What does not.
tests/e2e/ckpt/test_qwen3_4B_ckpt.pythen got past the missing module and hung inexecute("async_save")until the 90-minute step timeout:This is a second defect, independent of the strategy default. mcore's persistent checkpoint worker is a separate process, so the state dict crosses the boundary as pickled CUDA IPC handles.
torch_memory_saverbacks colocate-training allocations with the CUDA VMM APIs, which cannot export a legacy IPC handle, so_share_cuda_raises — and because it raises inside themultiprocessingqueue feeder thread, nothing propagates: the worker never receives the item and the trainer blocks forever.Megatron already fixes exactly this, in e338fedb4 ("stage async-ckpt tensors for CUDA IPC under torch_memory_saver"), whose commit message describes this failure verbatim. But the call is gated to one strategy:
The mcore worker has the same process architecture and the same need, so it should not be excluded. Everything downstream is already strategy-agnostic:
ipc_staging_scope()wrapsschedule_async_savefor anyasync_saveand no-ops unless restaging populated its slot list, andckpt_assume_constant_structuredefaults toFalse, so the restaging assertion holds.Suggested resolution
Two options, and I would like a maintainer's call:
async_strategy == "nvrx"from that guard inradixark/Megatron-LM@miles-main. Preserves async save on ROCm. It needs its own 8-GPU verification run.--async-savewithout--use-persistent-ckpt-worker. Sync save does not cross a process boundary —TorchDistSaveShardedStrategy.save()runsexecute_sync()in-process — and itsexecute("save")andexecute("load")phases passed in this very run. Slower, but correct and self-contained.I have not pushed either yet.
Not addressed here
The same nightly's 2-GPU job fails on
tests/e2e/fsdp/test_qwen3_0.6B_megatron_fsdp_align.py, which is unrelated and much older — it has failed 15 of the last 16 nightlies, going back to 2026-08-15, withtrain/ppo_kllanding between2e-9and2e-7againstcheck_kl's1e-9gate. That is a ROCm numerics question about whether the replayed Megatron forward reproduces the saved rollout logprobs, not a config one, and relaxing the threshold would just hide it.