Skip to content

feat: Add YaRN rope scaling support on Magatron-Bridge - #2188

Merged
yuki-97 merged 16 commits into
mainfrom
ruit/yarn_new
Apr 4, 2026
Merged

yuki-97 merged 16 commits into
mainfrom
ruit/yarn_new

Conversation

@RayenTian

@RayenTian RayenTian commented Apr 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR adds YaRN long-context support for Qwen3-0.6B Megatron workflows and introduces matching SFT/GRPO recipes and test coverage.

What changed

  • Added new 1n8g Megatron YaRN 64k recipes for:
    • SFT: sft-qwen3-0.6B-1n8g-megatron-yarn-64k
    • GRPO: grpo-qwen3-0.6B-1n8g-megatron-yarn-64k
  • Added matching test suite scripts for both recipes.
  • Added NemotronCascade2SFTMathDataset and registered it in the response dataset registry.
  • Updated Megatron setup to:
    • validate that YaRN rope_scaling fields are fully specified
    • force HF -> Megatron re-import when hf_config_overrides are present or when force_reimport_model=true
  • Updated GRPO setup so vLLM generation inherits the policy HF overrides, ensuring generation uses the same YaRN rope settings as training.
  • Added an Automodel guardrail to reject YaRN configs on the DTensor/Automodel backend, since YaRN is currently supported through the Megatron backend only.
  • Documented the force_reimport_model option in the training backend docs.

Issues

closes #1678

Test

SFT

image

GRPO

  • Yarn is working only when the seqlen exceeds the original model seqlen. Qwen2.5 and other models' original seqlens are about 128k or longer, which means we have to set seqlen to longer to enable yarn. And this'll lead to the long-tail issue being more notable, and further longer time_per_step.
    • For Qwen2.5-1.5B, seqlen=256k, some steps will take 1100+ seconds and 20 seconds for normal.
    • At the very beginning 20-30 steps, the max_gen_tokens_per_sample can reach 256K. This lead to long step time and high token_mult_prob_error. It will return to normal standard after few steps.
    • However, we cannot guarantee that setting max seqlen=256k will result in the rollout reaching 256k in every run.
    • Counterexample: The maximum is only 24k
image
  • Also tried Qwen3 models whose original context length is shorter (40k), while the original capabilities of Qwen3 models are too strong for the OpenMath dataset.
    • At the very beginning, the reward is 1.
    • As the experiences go on, the validation/accuracy drops. Tried on shorter seqlen with rope, also meet this issue. So it's not because of yarn.
    • Qwen3 tend to generate longer rollout, which lead to longer step time.
    • Qwen3-0.6B results
image
  • Also tried Qwen/Qwen2.5-Math-1.5B-Instruct, which has shorter original model seqlen(4k), so that we can get faster step time.
    • This model has been trained in previous fine-tuning to discourage the generation of longer seqlen. Even thought set max_model_len to 8K, It only in several steps can generate long seqlen to 8k.
    • It has strong initial mathematical capabilities, and the reward hardly increases.
    • Qwen/Qwen2.5-Math-1.5B-Instruct Results
image

All in all, finally choose qwen2.5-1.5B in the recipe. Its validation accuracy is increasing, and after the first 20-30 steps, it runs faster.

  • In VLLM, seqlen exceed the original model length is invalid if we don't open yarn. So we can not compare seqlen=256k results between yarn and rope. The result is the compare of seqlen= 128k + rope and seqlen=256k + yarn.

Final GRPO results(Qwen2.5-1.5B)

image

For max_gen_tokens_per_sample, at the very beginning step, it can reach to 256k/128k
image

Usage

  • You can potentially add a usage example below
# Add a code snippet demonstrating how to use this

Before your PR is "Ready for review"

Pre checks:

  • Make sure you read and followed Contributor guidelines
  • Did you write any new necessary tests?
  • Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

  • ...

@RayenTian
RayenTian requested review from a team as code owners April 2, 2026 02:13
@copy-pr-bot

copy-pr-bot Bot commented Apr 2, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added the Documentation Improvements or additions to documentation label Apr 2, 2026
@RayenTian RayenTian added the CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) label Apr 2, 2026
@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test 044a1b2

@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test 896a7b2

@terrykong terrykong left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor review — 2 findings.

[guideline: config-conventions] force_reimport_model is used in megatron/setup.py and present in all exemplar YAMLs, but not declared in the MegatronConfig TypedDict (nemo_rl/models/policy/__init__.py:199-251). Consider adding it following the force_overwrite_initial_ckpt pattern on line 224:

    # Force reimport of the HF model even if a converted checkpoint exists (default: False)
    force_reimport_model: NotRequired[bool]

Comment thread nemo_rl/models/megatron/setup.py Resolved
Comment thread nemo_rl/models/megatron/setup.py Resolved
Comment thread docs/design-docs/training-backends.md Resolved
Comment thread nemo_rl/algorithms/grpo.py Resolved
@RayenTian
RayenTian requested a review from a team as a code owner April 2, 2026 09:48
@RayenTian

Copy link
Copy Markdown
Contributor Author

Resolve comments. But the grpo recipe still under testing. The validation/accuracy is abnormal.

@RayenTian
RayenTian requested a review from a team as a code owner April 3, 2026 05:18
@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test abc6154

RayenTian added 15 commits April 3, 2026 19:06
…atron setups

Signed-off-by: ruit <ruit@nvidia.com>
…n in Automodel and Megatron setups

Signed-off-by: ruit <ruit@nvidia.com>
…ron and Yarn support

Signed-off-by: ruit <ruit@nvidia.com>
…tron and Yarn support

Signed-off-by: ruit <ruit@nvidia.com>
… 100, add prompts and generations settings. Adjust test scripts to reflect new max steps and validation metrics.

Signed-off-by: ruit <ruit@nvidia.com>
…t validation accuracy threshold in existing script.

Signed-off-by: ruit <ruit@nvidia.com>
… 'force_reconvert_from_hf' across various configuration files and update related documentation. Enhance model path validation to include hashed overrides for better checkpoint management.

Signed-off-by: ruit <ruit@nvidia.com>
Signed-off-by: ruit <ruit@nvidia.com>
…th Yarn support. Update nightly test suite to include new test script and adjust GPU hour limits in validation tests.

Signed-off-by: ruit <ruit@nvidia.com>
…r response datasets

Signed-off-by: ruit <ruit@nvidia.com>
…model settings

Signed-off-by: ruit <ruit@nvidia.com>
Signed-off-by: ruit <ruit@nvidia.com>
Signed-off-by: ruit <ruit@nvidia.com>
@RayenTian

Copy link
Copy Markdown
Contributor Author

/ok to test 9cc6bf3

This branch was previously deployed

1 inactive deployment
nemo-ci — 9cc6bf3a Deployed Apr 4, 2026 by copy-pr-bot[bot] via Docs_Tests #10766
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Can we apply yarn to Nemo-RL

3 participants