Skip to content

fix: add is_async to CheckpointingConfig TypedDict - #1991

Merged
yuki-97 merged 3 commits into
NVIDIA-NeMo:mainfrom
dmvevents:add-is-async-to-checkpointing-config
Mar 25, 2026
Merged

yuki-97 merged 3 commits into
NVIDIA-NeMo:mainfrom
dmvevents:add-is-async-to-checkpointing-config

Conversation

@dmvevents

@dmvevents dmvevents commented Feb 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • AutomodelCheckpointManager.init_checkpointer() already consumes is_async from config_updates (automodel_checkpoint.py:118)
  • AutomodelCheckpointManager.save_checkpoint() passes is_async through from the checkpointing_cfg dict (automodel_checkpoint.py:297)
  • However, the CheckpointingConfig(TypedDict) in checkpoint.py does not declare is_async as a field
  • This means Hydra config validation rejects +checkpointing.is_async=true as an unknown key, preventing users from enabling async checkpointing via config overrides

The fix adds a single NotRequired[bool] field to the CheckpointingConfig TypedDict, consistent with the existing pattern used by is_peft, save_consolidated, etc.

Test plan

  • Verify +checkpointing.is_async=true Hydra override is accepted without errors
  • Verify async checkpoint save/load works end-to-end with DTensor policy worker
  • Existing checkpoint tests continue to pass (field is NotRequired with default False)

Summary by CodeRabbit

  • New Features
    • Added optional asynchronous checkpoint configuration option to checkpoint settings.

@dmvevents
dmvevents requested a review from a team as a code owner February 19, 2026 15:20
@coderabbitai

coderabbitai Bot commented Feb 19, 2026 •

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

Added a new optional is_async field with a default value of False to the CheckpointingConfig TypedDict in the checkpoint utility module, extending the configuration schema without modifying existing fields or behavior.

Changes

Cohort / File(s) Summary
Checkpoint Configuration Schema
nemo_rl/utils/checkpoint.py
Added optional is_async: NotRequired[bool] field to CheckpointingConfig TypedDict to support asynchronous checkpoint operations.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Test Results For Major Changes ✅ Passed PR adds single-line optional TypedDict field enabling Hydra config overrides for existing parameter, with documented test plan covering acceptance, end-to-end functionality, and regression testing.
Title check ✅ Passed The title accurately reflects the main change: adding the is_async field to CheckpointingConfig TypedDict, which is the sole purpose of this PR.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
nemo_rl/utils/checkpoint.py (1)

38-52: 🛠️ Refactor suggestion | 🟠 Major

Document is_async in the class docstring and add it to exemplar YAML configurations.

The new field is not documented in the CheckpointingConfig docstring Attributes section. Additionally, exemplar YAML files under examples/configs/ that define checkpointing: blocks do not include the is_async configuration key. Per coding guidelines, new config keys must be documented and reflected in exemplar YAMLs.

✏️ Proposed docstring addition
     is_peft (bool): Whether the model uses PEFT.
+    is_async (bool): Whether to use asynchronous checkpointing. Defaults to False.
     """

Add is_async: false to each checkpointing: block in exemplar YAML files under examples/configs/.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@nemo_rl/utils/checkpoint.py` around lines 38 - 52, Update the
CheckpointingConfig class docstring to document the new is_async attribute
(e.g., is_async (bool): Whether checkpoint saving is performed asynchronously)
and ensure it appears in the Attributes section alongside enabled,
checkpoint_dir, etc.; then add is_async: false to every exemplar YAML
checkpointing: block under examples/configs/ so config examples reflect the new
key. Reference the CheckpointingConfig class and the checkpointing: blocks in
the example YAMLs when making these edits.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Outside diff comments:
In `@nemo_rl/utils/checkpoint.py`:
- Around line 38-52: Update the CheckpointingConfig class docstring to document
the new is_async attribute (e.g., is_async (bool): Whether checkpoint saving is
performed asynchronously) and ensure it appears in the Attributes section
alongside enabled, checkpoint_dir, etc.; then add is_async: false to every
exemplar YAML checkpointing: block under examples/configs/ so config examples
reflect the new key. Reference the CheckpointingConfig class and the
checkpointing: blocks in the example YAMLs when making these edits.

@chtruong814 chtruong814 added the needs-follow-up Issue needs follow-up label Feb 21, 2026
@ahmadki

ahmadki commented Feb 24, 2026

Copy link
Copy Markdown
Member

Thanks for the PR @dmvevents .
Do you mind singing the commit: https://github.com/NVIDIA-NeMo/RL/pull/1991/checks?check_run_id=64166609453 ?

@chtruong814 chtruong814 removed the needs-follow-up Issue needs follow-up label Feb 24, 2026
@dmvevents

Copy link
Copy Markdown
Contributor Author

Hi @ahmadki — apologies for the delay. Signing the commit and force-pushing now.

@chtruong814 chtruong814 added the needs-follow-up Issue needs follow-up label Mar 20, 2026
The is_async parameter is already consumed by AutomodelCheckpointManager
(automodel_checkpoint.py:118) and passed through in save_checkpoint
(automodel_checkpoint.py:297), but was missing from the CheckpointingConfig
TypedDict definition. This causes Hydra to reject
`+checkpointing.is_async=true` as an unknown key.

Signed-off-by: Anton Alexander <dmvevents@gmail.com>
@dmvevents
dmvevents force-pushed the add-is-async-to-checkpointing-config branch from c0393bb to 54e5646 Compare March 23, 2026 19:09
@copy-pr-bot

copy-pr-bot Bot commented Mar 23, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@dmvevents dmvevents changed the title Add is_async to CheckpointingConfig TypedDict fix: add is_async to CheckpointingConfig TypedDict Mar 23, 2026
@chtruong814 chtruong814 removed the needs-follow-up Issue needs follow-up label Mar 23, 2026
@dmvevents

Copy link
Copy Markdown
Contributor Author

@ahmadki — commit is now signed with the correct DCO sign-off and PR title updated to pass semantic-pull-request. All CI checks are green. Ready for re-review when you get a chance.

@yuki-97 yuki-97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hi @dmvevents , thanks for adding this!

@yuki-97 yuki-97 added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label Mar 25, 2026
@yuki-97

yuki-97 commented Mar 25, 2026

Copy link
Copy Markdown
Contributor

/ok to test af53275

@chtruong814

Copy link
Copy Markdown
Contributor

/ok to test 83d9765

@yuki-97
yuki-97 merged commit b0da493 into NVIDIA-NeMo:main Mar 25, 2026
47 of 49 checks passed
yuekaizhang added a commit to yuekaizhang/RL that referenced this pull request Aug 25, 2026
…le3 co-training recipes at 10240 context

MoE sibling of the Qwen3-8B math-baseline recipe: EP=32 shards the
policy's 128 experts across all GPUs (dtensor TP=1), vLLM generation runs
TP=4. Both recipes run at 10240-token context (2048 prompt + 8192
response), not this repo's usual 20480: eagle3 draft co-training against
RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.eagle3 OOMs at 20480 on
this layout even after chunking the drafter's lm_head/KL (the base memory
budget is too tight at that length), and the no-draft baseline gets a
matching 10240 sibling so the two stay comparable at the same sequence
length.

Both also switch checkpointing to synchronous dcp.save (is_async: false):
DTensorPolicyWorkerV2's default async checkpoint path (dcp.async_save's
per-rank Gloo rendezvous) has failed intermittently on this 4-node/32-rank
layout at the first save_period boundary (see NVIDIA-NeMo#1991, already
merged, which made checkpointing.is_async a valid config field).

eagle3-10k is the first recipe pairing draft co-training with EP-sharded
policy training in this repo -- treat as unvalidated until it completes a
smoke run.

Signed-off-by: Yuekai Zhang <zhangyuekai@foxmail.com>

This branch was previously deployed

1 inactive deployment
nemo-ci — 83d97654 Deployed Mar 25, 2026 by copy-pr-bot[bot] via fast_L1_Functional_Tests_GPU #10635
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) community-request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants