Skip to content

feat(draft): add DSpark tiled objectives - #3726

Merged
yuki-97 merged 112 commits into
NVIDIA-NeMo:mainfrom
seonjinn:seonjinn/draft-cotrain-dspark-objective-v3-20260820
Nov 11, 2026
Merged

yuki-97 merged 112 commits into
NVIDIA-NeMo:mainfrom
seonjinn:seonjinn/draft-cotrain-dspark-objective-v3-20260820

Conversation

@seonjinn

@seonjinn seonjinn commented Aug 20, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Adds tiled hard-CE, total-variation, and confidence objectives for DSpark.
  • Trains hard CE on sampled draft labels and applies position weights to numerators and denominators.
  • Keeps target logits and the live output head stop-gradient.
  • Supports split vocabularies, bounded-memory chunking, and synchronized TP contract failures.

Scope

Covers DSpark objectives, heads, providers, and distributed objective tests. Training-loop configuration is handled downstream.

  • Unit-test coverage fix: pyproject.toml sets source_pkgs, tests/run_unit.sh passes --cov-config, and tests/unit/conftest.py keeps the session coverage total. tests/unit/test_unit_coverage_lifecycle.py tests this.

Validation

  • Merged with main d3d0cc7586. Review fixes in c6299c47c0.
  • Local CPU: tests/unit/algorithms/test_dspark_objective.py 51 passed. GPU TP/DP tests not run locally. CI is pending.

Prior focused evidence

  • OCI 6392604: 42 focused tests passed. The local objective suite passed 32 tests; earlier TP/DP and real-MCore checkpoint coverage also passed on 4×GB200.

Dependencies

Depends on #3703 and #3711.

Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
…n-dspark-objective-v3-fixes

Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
@seonjinn

Copy link
Copy Markdown
Contributor Author

/ok to test c6299c4

…n-dspark-objective-v3-fixes

Signed-off-by: seonjinn <sna@nvidia.com>
@seonjinn

Copy link
Copy Markdown
Contributor Author

/ok to test c39e31e

…n-dspark-objective-v3-20260820

Signed-off-by: seonjinn <sna@nvidia.com>
Block draft co-training can train at the baseline TP4 layout. Run the
DSpark objective and provider TP tests at TP2 and TP4 to cover the
split-vocab paths at both sizes.

Signed-off-by: seonjinn <sna@nvidia.com>
@seonjinn

Copy link
Copy Markdown
Contributor Author

/ok to test c608fd4

copy-pr-bot Bot pushed a commit that referenced this pull request Oct 22, 2026
Signed-off-by: seonjinn <sna@nvidia.com>
yuekaizhang
yuekaizhang previously approved these changes Oct 28, 2026

@yuekaizhang yuekaizhang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the draft co-training refit/transport stack across #3723/#3726/#3744/#3750/#3757. All findings from this review are resolved (see individual threads), CI is green. Approving.

Signed-off-by: seonjinn <sna@nvidia.com>

# Conflicts:
#	docs/index.md
@seonjinn

seonjinn commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test e96a75f

@yuki-97 yuki-97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Merge blockers — please rebase on main; the PR is 22 commits behind and the "Check if PR branch is up to date" job allows at most 10.

Thanks for adding tiled DSpark CE, TV, and confidence objectives with a stop-gradient target head.

The reported four-GPU GB200 checks and extensive self-review are appreciated; the posted results make the validation scope clear.

Scope: DSpark objectives, the provider, documentation, and shared coverage changes, at e96a75f.

Checked:

  • Loss math — global normalization and TP-summed hidden and Markov gradients match the dense-reference contract.
  • Consumer integration — #3757 consumes the combined tensors through the common draft-loss wrapper; the dedicated component interfaces have only test consumers.
  • Validation — current-head CI logs confirm objective, provider, coverage lifecycle, TP2/DP2, and TP2 checkpoint cases passed; TP4 cases skip on two-GPU runners. Downstream e2e runs remain author-reported.

Remaining comments are on:

  • Removing the dedicated statistics interfaces that have only test consumers while preserving validation and gradient coverage.
  • Removing the shared coverage changes from this PR and landing any needed fixes separately, as previously requested on #3715 and #3711.

nit

  • Shorten the confidence-path comment and remove its unsupported future guarantee.
  • Add the downstream integration pointer to the provider factory.

Generated by Codex

Comment thread nemo_rl/algorithms/loss/dspark.py Resolved
Comment thread pyproject.toml Resolved
@terrykong terrykong added CI:L1 Run doctests, unit tests, and functional tests and removed CI:Lfast Runs a fast test suite and re-use nightly `main` container (but sync dependencies to PRs version) labels Nov 5, 2026
…n-dspark-objective-v3-fixes

Signed-off-by: seonjinn <sna@nvidia.com>
Signed-off-by: seonjinn <sna@nvidia.com>
@seonjinn

seonjinn commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

/ok to test 7fe724d

seonjinn added a commit to seonjinn/RL that referenced this pull request Nov 7, 2026
…rf-20261020

Signed-off-by: seonjinn <sna@nvidia.com>
seonjinn added a commit to seonjinn/RL that referenced this pull request Nov 7, 2026
Signed-off-by: seonjinn <sna@nvidia.com>
@yuki-97
yuki-97 enabled auto-merge (squash) November 10, 2026 13:12
@yuki-97

yuki-97 commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

/ok to test 8bcb841

@yuki-97
yuki-97 merged commit e6aae2a into NVIDIA-NeMo:main Nov 11, 2026
94 checks passed
jthomson04 added a commit to jthomson04/RL that referenced this pull request Nov 12, 2026
Brings NVIDIA-NeMo#3726 (DSpark tiled objectives), NVIDIA-NeMo#4882 (reward_shaping on the
SingleController path) and NVIDIA-NeMo#5130 (in-loss sequence filtering for PPO).
No conflicts.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Signed-off-by: jthomson04 <jwillthomson19@gmail.com>
seonjinn added a commit to seonjinn/RL that referenced this pull request Nov 13, 2026
Signed-off-by: seonjinn <sna@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI:L1 Run doctests, unit tests, and functional tests Documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants