Skip to content

feat(decisions): validate native scores and answer-position tokenization - #15850

Merged
biswapanda merged 7 commits into
mainfrom
bis/decisions-d2-native
Oct 27, 2026
Merged

biswapanda merged 7 commits into
mainfrom
bis/decisions-d2-native

Conversation

@biswapanda

@biswapanda biswapanda commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Add the exact native SGLang prefill-only candidate-scoring request envelope and strict terminal response validation.
  • Verify complete candidate identity/cardinality, explicit zero completion accounting, and rejection of malformed, partial, reordered, decoded, or aborted results.
  • Verify distinct single-token answer labels against the actual rendered answer position, reject incompatible reasoning/template states, and bound tokenizer work.

Stack

Dynamo review order: D1: mocker → D2: native scoring → D3: discovery → D4: admission → D5: execution → D6a: HTTP → D6b: OpenAPI → D7: qualification. Each PR targets its predecessor; merge from the base upward and update dependent bases as prerequisites land.

D2 of the Dynamo decision API stack; depends on D1: mocker scoring. Base: bis/decisions-d1-mocker.

This slice introduces native transport and tokenizer primitives only. Discovery-bound preparation and branch execution follow in later slices; public routes remain absent. D2 itself does not require the new frontend crate, so it can be reviewed independently of that cross-repository release boundary.

Related: umbrella DEP #15843 and its PR dependency plan.

Validation

Current stack rebased onto main acef8afe0a5b6c7ae9acec3d1ad4fd76234efc5f, with every PR targeting its latest predecessor. Signed rebase commits supersede the earlier published heads.

Fresh integrated-stack validation and complete GPU pytest output ran at tip 876e2980df44ac3ec16d3079965e1c58039e5cd3: both committed native SGLang tests passed on stock 0.5.22/GB200 with pinned Qwen3-0.6B; four official OpenAI/TypeSafe client/wire cases passed separately against the live worker. The incompatible 43-choice Jev probe returned the new actionable HTTP 422 diagnostic. These are integrated-stack results, not a claim that lower slices expose public routes independently.

Final-source validation also passed 294 mocker tests, 51 System One tests, 52 frontend service tests, nine live CPU HTTP tests, focused routing/protocol/lifecycle/OpenAPI and Python regressions, strict LLM all-target Clippy without default features, formatting/pre-commit, and locked metadata. Documentation configuration and broken-link checks passed. Counts overlap; new-head remote CI is tracked separately. This does not qualify the 27B model, model quality, performance/scaling, or physical GPU cancellation settlement.

Historical validation before this main rebase

Rebased onto main 02da7f44861b65758b960770e0763e9faa8e4ca4. D3 and later use crates.io dynamo-decisions =0.1.0 with dynamo-protocols =8.2.0; obsolete Git dependency/source patches are removed. The prior unpublished-crate prerequisite is no longer applicable.

Validation below is integrated-stack evidence, not a claim that lower slices already expose public HTTP routes.

  • Locked Cargo metadata passed for the root, Python-binding and KVBM-binding workspaces; published decision-core tests: 40 passed.
  • Fresh Python binding build passed. Ten CPU HTTP/helper tests and ten manual-marker policy tests passed; malformed content types and empty questions are rejected before worker inference.
  • Nine wire/official synchronous SDK checks passed against real cached Qwen/Qwen3.8-27B revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, SGLang 0.5.19, and the final combined source 2f90adc5360cbdd5de6e5d78e9fb150cb5513595; independent-native parity was deselected.
  • Rust formatting and whitespace checks passed. A broader local Rust test rebuild hit disk exhaustion and is not counted as a passing test run; focused reruns and actual current-head remote CI remain separately tracked.

Only OpenAI /v1/decisions and Jev /v1/systemone are public decision contracts. No native SGLang wire format or nvext.format selector is supported. No new claim of independent numerical parity, Decision Index quality, full client-language coverage or physical GPU settlement is made.

Historical validation before the registry integration

The following records concern earlier source revisions and are not current-head CI evidence.

  • Native scoring protocol integration tests: five passed.
  • Tokenizer/preflight tests: two passed.
  • Tests run against the stack based on current main 3213fe6f03.
  • Rust formatting and whitespace checks: passed.
  • Standalone library/test Clippy on this branch: passed with warnings denied. The unit-tested private preflight entry point has an exact-item lint expectation until the execution layer consumes it; that later slice removes the expectation.

These tests validate transport/preprocessing contracts, not broad model quality or physical GPU cancellation settlement. GPU numerical parity and official-client fixtures are retained in the qualification slice.

Summary by CodeRabbit

  • New Features
    • Added native SystemOne candidate scoring support, including requests for scores across candidate labels and results returned in the requested order.
    • Added prompt checks for reasoning mode, prompt limits, and candidate-label tokenization before scoring.
    • Invalid prompts, candidate scores, and scoring responses are now rejected rather than accepted as valid results.

Implementation tracking: Part of #15961. This is a non-closing link to the decision-model delivery roadmap; the umbrella DEP remains open.

@devin-ai-integration

devin-ai-integration Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 40454960805 (attempt 1) on 555535744d

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 40454960805. Updated on every full-CI run of this PR.

@biswapanda
biswapanda added this pull request to stack #15885 October 8, 2026 06:23
@biswapanda
biswapanda marked this pull request as ready for review October 8, 2026 06:25
@biswapanda
biswapanda requested review from a team as code owners October 8, 2026 06:25

Copy link
Copy Markdown
Contributor Author

/ok to test a0b6ee5

@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: f75c2251-8bab-41af-9b89-af30cfb1e6be
📥 Commits

Reviewing files that changed from the base of the PR and between 50dc85b and a0b6ee5.

📒 Files selected for processing (5)
  • lib/llm/src/preprocessor.rs
  • lib/llm/src/protocols.rs
  • lib/llm/src/protocols/systemone/mod.rs
  • lib/llm/src/protocols/systemone/native.rs
  • lib/llm/tests/systemone_protocol.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 10 remain after this review.

Walkthrough

The change adds System One prompt validation and native score request and response handling. Prompt rendering checks reasoning behavior, tokenization consistency, encoded-byte limits, prompt-token limits, and candidate-label token IDs. The protocol builds native scoring requests and parses candidate scores.

Changes

System One scoring

Layer / File(s) Summary
Prompt validation
lib/llm/src/preprocessor.rs, lib/llm/tests/*
Adds render_systemone_question and internal limit errors. The method checks reasoning behavior, tokenization consistency, encoded-byte and prompt-token limits, and candidate-label token IDs. Tests check validation order and reasoning-parser rejection.
Native score requests and parsing
lib/llm/src/protocols.rs, lib/llm/src/protocols/systemone/*, lib/llm/tests/systemone_protocol.rs
Publicly declares the systemone protocol and exposes native request-building and score-parsing functions. Request construction validates its inputs and builds the scoring envelope. Parsing validates the response and returns scores in expected-token order. Tests cover request construction and score parsing.

Priority: Low

Estimated code review effort: 3 (Moderate) | ~25 minutes

Merge Risk: Minimal · up to a0b6e

No actionable issue in this slice was established; it is mergeable after normal checks.

Pre-merge checks: Passed 4
Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Title check Passed The title clearly identifies the main changes: native score validation and answer-position tokenization.
Description check Passed The description explains the changes, stack context, validation, and related issues. It does not use the template’s exact section headings or name specific files for reviewer focus, but it is mostly c…
Full details: Description check

Explanation

The description explains the changes, stack context, validation, and related issues. It does not use the template’s exact section headings or name specific files for reviewer focus, but it is mostly complete.

✨ Finishing Touches: 💡 1
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Comment thread lib/llm/tests/systemone_protocol.rs Resolved
Comment thread lib/llm/tests/systemone_protocol.rs Resolved
@biswapanda
biswapanda marked this pull request as draft October 20, 2026 01:01
@biswapanda
biswapanda force-pushed the bis/decisions-d2-native branch from a0b6ee5 to 7aa8afa Compare October 20, 2026 01:02
@biswapanda
biswapanda marked this pull request as ready for review October 20, 2026 17:23
@biswapanda
biswapanda force-pushed the bis/decisions-d2-native branch from 7aa8afa to 46a9813 Compare October 20, 2026 19:58
@biswapanda biswapanda self-assigned this Oct 20, 2026

@luckyq luckyq left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Response validation looks right: order/cardinality enforced against the request, non-null token text rejected, output_ids:[42]+completion_tokens:0 and []+1 both rejected, D1 mocker envelope compatible. Approving; please follow up on test coverage:

  • lib/llm/src/preprocessor.rs:3049-3135: the label-tokenization checks (encoded.len() == prompt_ids.len() + 1, prefix equality, distinct label ids) have no test. Both new tests exit before the label loop and use a byte-per-token stub, so deleting those checks passes the suite. A positive + negative case with a real tokenizer (e.g. mock-llama, including a leading-space label) would cover the BPE boundary behaviour the feature depends on.
  • preprocessor.rs:3080: the reasoning-start check uses reasoning_parser directly; the chat path goes through configured_family(tool_call_parser, reasoning_parser). With only tool_call_parser=gemma4/kimi_k3/minimax_m3 set, the generic <think> check misses those families' markers.
  • preprocessor.rs:3107: a round-trip mismatch caused by user content containing special-token text is classified as SystemOneModelIncompatible, i.e. a per-request problem reported as a model-level one.

@biswapanda

Copy link
Copy Markdown
Contributor Author

fixed in 3c1269d. Review follow-up: real TinyLlama BPE fixtures now exercise leading-space labels, multi-token labels, duplicate normalized IDs, and append-only prefix mismatch. User-content round-trip failures are classified as invalid requests rather than model incompatibility. Reasoning-family detection now accounts for tool-parser configuration; 30e5e76 adds the narrowly scoped MiniMax opener follow-up without changing chat preprocessing. Targeted protocol and preprocessing regressions passed.

@jh-nv jh-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 30e5e76. Nothing blocking. Approving.

@biswapanda
biswapanda force-pushed the bis/decisions-d2-native branch from 30e5e76 to 5555357 Compare October 27, 2026 05:22
Base automatically changed from bis/decisions-d1-mocker to main October 27, 2026 16:27
@biswapanda
biswapanda requested a review from a team as a code owner October 27, 2026 16:27
@biswapanda
biswapanda force-pushed the bis/decisions-d2-native branch from 5555357 to c6ec85a Compare October 27, 2026 16:27
@biswapanda
biswapanda removed the request for review from a team October 27, 2026 16:27
@biswapanda
biswapanda enabled stacked auto-merge (squash) October 27, 2026 16:28
@biswapanda
biswapanda disabled stacked auto-merge October 27, 2026 16:55
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
@biswapanda
biswapanda force-pushed the bis/decisions-d2-native branch from c6ec85a to 31bf442 Compare October 27, 2026 18:01

@krishung5 krishung5 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@biswapanda
biswapanda merged commit 0b53b9b into main Oct 27, 2026
210 of 219 checks passed
@biswapanda
biswapanda deleted the bis/decisions-d2-native branch October 27, 2026 21:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feat frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants