Skip to content

[AMD] Fall back to mcore async checkpointing when NVRx is unavailable - #2816

Draft
michaelzhang-ai wants to merge 1 commit into
radixark:mainfrom
michaelzhang-ai:cursor/rocm-async-save-mcore-fallback-3d8b
Draft

michaelzhang-ai wants to merge 1 commit into
radixark:mainfrom
michaelzhang-ai:cursor/rocm-async-save-mcore-fallback-3d8b

Conversation

@michaelzhang-ai

@michaelzhang-ai michaelzhang-ai commented Aug 30, 2026 •

Copy link
Copy Markdown
Contributor

Warning

Do not merge as-is. Hardware verification showed this change is necessary but not sufficient: it removes the ModuleNotFoundError but the run then hangs for 90 minutes on a second, independent ROCm defect. Merging only this would trade a fast, legible failure for a silent hang, which is worse. See Verification — the companion Megatron change is identified precisely and this PR should land with it, or be re-scoped.

Summary

--async-save is broken on every ROCm image. Megatron defaults --async-strategy to "nvrx", which hard-requires nvidia-resiliency-ext; docker/Dockerfile.rocm deliberately strips that CUDA-only package out of requirements.txt:

https://github.com/radixark/miles/blob/main/docker/Dockerfile.rocm#L167

So the run parses fine, trains fine, and then dies at the first checkpoint write:

ray::MegatronTrainRayActor.save_model()
  File "/root/Megatron-LM/megatron/core/dist_checkpointing/strategies/torch.py", line 1169, in get_async_strategy
    raise ModuleNotFoundError(
ModuleNotFoundError: A compatible `nvidia-resiliency-ext` installation is required for
`async_strategy="nvrx"`. Please install it or set `async_strategy` to `mcore`.

set_default_megatron_args now probes for the NVRx async-checkpoint API and falls back to Megatron's in-tree "mcore" saver when it is missing. CUDA images ship the package, so they keep using nvrx and are untouched.

How this surfaced

tests/e2e/ckpt/test_qwen3_4B_ckpt.py in the MI350 nightly, which runs execute("save") → execute("load") → execute("async_save") → execute("load"). The first nightlies to run on a post-bump image all failed on the third step:

nightly miles image test_qwen3_4B_ckpt.py
2026-08-26 miles-rocm720-mi35x-20260824 pass
2026-08-27 miles-rocm720-mi35x-20260824 pass
2026-08-28 miles-rocm720-mi35x-20260828 fail
2026-08-29 miles-rocm720-mi35x-20260829 fail

Failing run: sgl-project/sglang actions/runs/33265909152, job nightly-stage-c-8-gpu-mi350.

The 8/25–8/27 nightlies reused the stale -20260824 tag because the ROCm image builds for those days had failed, which is why a change that landed on 8/24 only turned the nightly red on 8/28.

Why the bump changed behaviour

async_strategy did not exist before the Megatron bump in #2673. Counting the Megatron argument dumps in each nightly's 8-GPU log makes the transition explicit — same test set, same 26 training invocations, exactly one of which asks for async save:

image 20260824 (8/27, passed) image 20260829 (8/29, failed)
async_save ... False 25 24
async_save ... True 1 1
async_strategy ... mcore — not printed — 24
async_strategy ... nvrx — not printed — 1

On the old image the option is absent entirely, so async save always used the in-tree writer. On the new image the 24 sync-save runs get mcore (Megatron's own validation rewrites nvrx → mcore whenever async_save is off), and the single async-save run keeps the nvrx default and hits the missing module.

Verification

Run on real MI350 hardware against the nightly's own image (rocm/sgl-dev:miles-rocm720-mi35x-20260829, torch 2.9.1+rocm7.2.0, hip 7.2.26015), by patching the in-image /root/miles from refs/pull/2816/head and measuring before and after in the same container: sgl-project/sglang actions/runs/33305121208.

What works. The stock image reproduces the nightly's exception exactly, and the patch clears it:

stock image with this PR
nvidia_resiliency_ext present False False
resolved async_strategy nvrx mcore
backend import ModuleNotFoundError: A compatible nvidia-resiliency-ext installation is required... loaded mcore

All five new unit tests also ran in that container against the real Megatron rather than being skipped by their importorskip. In the end-to-end run, all four Megatron invocations reported async_strategy ... mcore, including the single async_save=True one — before the patch that one was nvrx.

What does not. tests/e2e/ckpt/test_qwen3_4B_ckpt.py then got past the missing module and hung in execute("async_save") until the 90-minute step timeout:

[actor_cell0_rank0] torch.py:763 - MCore's async save is deprecated ...
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 244, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/sgl-workspace/sglang/python/sglang/srt/utils/patch_torch.py", line 73, in _reduce_tensor_modified
    output_fn, output_args = reductions._reduce_tensor_original(*args, **kwargs)
  File ".../torch/multiprocessing/reductions.py", line 354, in reduce_tensor
    ) = storage._share_cuda_()
torch.AcceleratorError: HIP error: invalid argument

This is a second defect, independent of the strategy default. mcore's persistent checkpoint worker is a separate process, so the state dict crosses the boundary as pickled CUDA IPC handles. torch_memory_saver backs colocate-training allocations with the CUDA VMM APIs, which cannot export a legacy IPC handle, so _share_cuda_ raises — and because it raises inside the multiprocessing queue feeder thread, nothing propagates: the worker never receives the item and the trainer blocks forever.

Megatron already fixes exactly this, in e338fedb4 ("stage async-ckpt tensors for CUDA IPC under torch_memory_saver"), whose commit message describes this failure verbatim. But the call is gated to one strategy:

# megatron/core/dist_checkpointing/strategies/torch.py:901
if async_strategy == "nvrx" and not _gpu_tensors_are_ipc_shareable(writer):
    _restage_gpu_tensors_ipc_shareable(writer)

The mcore worker has the same process architecture and the same need, so it should not be excluded. Everything downstream is already strategy-agnostic: ipc_staging_scope() wraps schedule_async_save for any async_save and no-ops unless restaging populated its slot list, and ckpt_assume_constant_structure defaults to False, so the restaging assertion holds.

Suggested resolution

Two options, and I would like a maintainer's call:

  1. Land this together with dropping async_strategy == "nvrx" from that guard in radixark/Megatron-LM@miles-main. Preserves async save on ROCm. It needs its own 8-GPU verification run.
  2. Re-scope this PR to disable async save on ROCm rather than downgrade it, matching Megatron's own precedent for --async-save without --use-persistent-ckpt-worker. Sync save does not cross a process boundary — TorchDistSaveShardedStrategy.save() runs execute_sync() in-process — and its execute("save") and execute("load") phases passed in this very run. Slower, but correct and self-contained.

I have not pushed either yet.

Not addressed here

The same nightly's 2-GPU job fails on tests/e2e/fsdp/test_qwen3_0.6B_megatron_fsdp_align.py, which is unrelated and much older — it has failed 15 of the last 16 nightlies, going back to 2026-08-15, with train/ppo_kl landing between 2e-9 and 2e-7 against check_kl's 1e-9 gate. That is a ROCm numerics question about whether the replayed Megatron forward reproduces the saved rollout logprobs, not a config one, and relaxing the threshold would just hide it.

Megatron defaults --async-strategy to "nvrx", which hard-requires
nvidia-resiliency-ext. docker/Dockerfile.rocm strips that CUDA-only package
from requirements.txt, so on ROCm every --async-save run reached the first
checkpoint write and died with:

  ModuleNotFoundError: A compatible `nvidia-resiliency-ext` installation is
  required for `async_strategy="nvrx"`

Probe for the NVRx async-checkpoint API in set_default_megatron_args and drop
back to Megatron's in-tree "mcore" saver when it is missing. CUDA images ship
the package, so they keep using nvrx unchanged.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants