Repository navigation
fix: add is_async to CheckpointingConfig TypedDict - #1991
Conversation
📝 WalkthroughWalkthroughAdded a new optional Changes
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~2 minutes 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
nemo_rl/utils/checkpoint.py (1)
38-52: 🛠️ Refactor suggestion | 🟠 MajorDocument
is_asyncin the class docstring and add it to exemplar YAML configurations.The new field is not documented in the
CheckpointingConfigdocstringAttributessection. Additionally, exemplar YAML files underexamples/configs/that definecheckpointing:blocks do not include theis_asyncconfiguration key. Per coding guidelines, new config keys must be documented and reflected in exemplar YAMLs.✏️ Proposed docstring addition
is_peft (bool): Whether the model uses PEFT. + is_async (bool): Whether to use asynchronous checkpointing. Defaults to False. """Add
is_async: falseto eachcheckpointing:block in exemplar YAML files underexamples/configs/.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@nemo_rl/utils/checkpoint.py` around lines 38 - 52, Update the CheckpointingConfig class docstring to document the new is_async attribute (e.g., is_async (bool): Whether checkpoint saving is performed asynchronously) and ensure it appears in the Attributes section alongside enabled, checkpoint_dir, etc.; then add is_async: false to every exemplar YAML checkpointing: block under examples/configs/ so config examples reflect the new key. Reference the CheckpointingConfig class and the checkpointing: blocks in the example YAMLs when making these edits.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Outside diff comments:
In `@nemo_rl/utils/checkpoint.py`:
- Around line 38-52: Update the CheckpointingConfig class docstring to document
the new is_async attribute (e.g., is_async (bool): Whether checkpoint saving is
performed asynchronously) and ensure it appears in the Attributes section
alongside enabled, checkpoint_dir, etc.; then add is_async: false to every
exemplar YAML checkpointing: block under examples/configs/ so config examples
reflect the new key. Reference the CheckpointingConfig class and the
checkpointing: blocks in the example YAMLs when making these edits.
|
Thanks for the PR @dmvevents . |
|
Hi @ahmadki — apologies for the delay. Signing the commit and force-pushing now. |
The is_async parameter is already consumed by AutomodelCheckpointManager (automodel_checkpoint.py:118) and passed through in save_checkpoint (automodel_checkpoint.py:297), but was missing from the CheckpointingConfig TypedDict definition. This causes Hydra to reject `+checkpointing.is_async=true` as an unknown key. Signed-off-by: Anton Alexander <dmvevents@gmail.com>
c0393bb to
54e5646
Compare
|
@ahmadki — commit is now signed with the correct DCO sign-off and PR title updated to pass semantic-pull-request. All CI checks are green. Ready for re-review when you get a chance. |
yuki-97
left a comment
There was a problem hiding this comment.
hi @dmvevents , thanks for adding this!
|
/ok to test af53275 |
|
/ok to test 83d9765 |
…le3 co-training recipes at 10240 context MoE sibling of the Qwen3-8B math-baseline recipe: EP=32 shards the policy's 128 experts across all GPUs (dtensor TP=1), vLLM generation runs TP=4. Both recipes run at 10240-token context (2048 prompt + 8192 response), not this repo's usual 20480: eagle3 draft co-training against RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.eagle3 OOMs at 20480 on this layout even after chunking the drafter's lm_head/KL (the base memory budget is too tight at that length), and the no-draft baseline gets a matching 10240 sibling so the two stay comparable at the same sequence length. Both also switch checkpointing to synchronous dcp.save (is_async: false): DTensorPolicyWorkerV2's default async checkpoint path (dcp.async_save's per-rank Gloo rendezvous) has failed intermittently on this 4-node/32-rank layout at the first save_period boundary (see NVIDIA-NeMo#1991, already merged, which made checkpointing.is_async a valid config field). eagle3-10k is the first recipe pairing draft co-training with EP-sharded policy training in this repo -- treat as unvalidated until it completes a smoke run. Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>
Summary
AutomodelCheckpointManager.init_checkpointer()already consumesis_asyncfromconfig_updates(automodel_checkpoint.py:118)AutomodelCheckpointManager.save_checkpoint()passesis_asyncthrough from thecheckpointing_cfgdict (automodel_checkpoint.py:297)CheckpointingConfig(TypedDict)incheckpoint.pydoes not declareis_asyncas a field+checkpointing.is_async=trueas an unknown key, preventing users from enabling async checkpointing via config overridesThe fix adds a single
NotRequired[bool]field to theCheckpointingConfigTypedDict, consistent with the existing pattern used byis_peft,save_consolidated, etc.Test plan
+checkpointing.is_async=trueHydra override is accepted without errorsNotRequiredwith defaultFalse)Summary by CodeRabbit