Skip to content

test(decisions): add explicit native and SDK qualification profiles - #15857

Merged
biswapanda merged 12 commits into
mainfrom
bis/decisions-d7-qualification
Oct 30, 2026
Merged

biswapanda merged 12 commits into
mainfrom
bis/decisions-d7-qualification

Conversation

@biswapanda

@biswapanda biswapanda commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Preserve independent native-SGLang numerical parity and scoring/chat-coexistence fixtures using a small qualification model.
  • Add opt-in tests against an explicitly provisioned server for both Phase 1 wire contracts and official OpenAI/TypeSafe Python synchronous SDK evaluation, errors, and headers.
  • Require explicit model/server selection and keep large Qwen3.8-27B provisioning outside shared CI. SDK requirements remain optional unless explicitly enforced; the shared test environment's SDK pin is unchanged.
  • Gate the engine fixture explicitly to the qualified SGLang 0.5.19 and 0.5.22 profiles; other runtimes and execution configurations require separate qualification.

Phase 1 scope

The integrated stack supports only OpenAI POST /v1/decisions and Jev/TypeSafe POST /v1/systemone. Every top-level nvext envelope is rejected, including {} and explicit "oai". Public SGLang-native Decisions request/response types, adapters, projection and selector support are removed. SGLang /generate scoring remains an internal execution primitive, not a third public contract. Optional native-format support is deferred to DEP #15960. This scope supersedes the earlier three-contract implementation description.

Stack

Dynamo review order: D1: mocker → D2: native scoring → D3: discovery → D4: admission → D5: execution → D6a: HTTP → D6b: OpenAPI → D7: qualification. Each PR targets its predecessor; merge from the base upward and update dependent bases as prerequisites land.

D7, the final Dynamo qualification-fixture slice. Base: bis/decisions-d6b-observability.

Related: umbrella DEP #15843 and its dependency plan.

Reproduction

Run within the repository's installed Dynamo test environment, with an explicitly provisioned trusted/local server. DYNAMO_DECISION_TEST_URL is the origin without /v1, and the model value must match its registered name. This test entry point does not provision GPUs or download the large model.

DYNAMO_DECISION_TEST_URL=http://localhost:8000 \
DYNAMO_DECISION_TEST_MODEL=Qwen/Qwen3.8-27B \
DYNAMO_DECISION_REQUIRE_SDKS=1 \
python -m pytest -q tests/serve/test_decisions_external.py \
  -k 'not native_sglang_numeric_parity' \
  --junitxml=decision-sdk-results.xml

The previously qualified optional clients are openai==3.26.0 and typesafe-sdk==0.7.2; install them in a dedicated qualification environment, not by changing the shared test dependency pin. For independent native parity, also set a distinct DYNAMO_DECISION_NATIVE_URL and DYNAMO_DECISION_TOKENIZER_PATH to the exact pinned local tokenizer. Raw HTTP/parity calls currently target trusted/local unauthenticated endpoints; SDK key configuration alone does not qualify an authenticated gateway.

The small-model launch fixture additionally requires DYNAMO_DECISION_RUN_PINNED_SGLANG=1 and a qualified SGLang 0.5.19 or 0.5.22 runtime. Without explicit opt-in, collection deselects it rather than launching a GPU workload. This does not qualify arbitrary newer runtimes or configurations.

Validation

The stack is rebased onto main 532616d12ddea4d9ebce30fbcb67807cc02b63e0, with every remaining PR based on its updated predecessor. It preserves main's released dynamo-protocols =8.4.0 and dynamo-parsers =10.2.2 alongside dynamo-decisions =0.1.0. Validation below is historical evidence at the source revisions explicitly identified; the review-update comment records final-head validation.

Recorded integrated-stack validation and complete GPU pytest output ran at tip 876e2980df44ac3ec16d3079965e1c58039e5cd3: both committed native SGLang tests passed on stock 0.5.22/GB200 with pinned Qwen3-0.6B; four official OpenAI/TypeSafe client/wire cases passed separately against the live worker. The incompatible 43-choice Jev probe returned the new actionable HTTP 422 diagnostic. These are integrated-stack results, not a claim that lower slices expose public routes independently.

Prior recorded source validation also passed 294 mocker tests, 51 System One tests, 52 frontend service tests, nine live CPU HTTP tests, focused routing/protocol/lifecycle/OpenAPI and Python regressions, strict LLM all-target Clippy without default features, formatting/pre-commit, and locked metadata. Documentation configuration and broken-link checks passed. Counts overlap; new-head remote CI is tracked separately. This does not qualify the 27B model, model quality, performance/scaling, or physical GPU cancellation settlement.

Historical validation before this main rebase

Rebased onto main 02da7f44861b65758b960770e0763e9faa8e4ca4. D3 and later use crates.io dynamo-decisions =0.1.0 with dynamo-protocols =8.2.0; obsolete Git dependency/source patches are removed. The prior unpublished-crate prerequisite is no longer applicable.

Validation below is integrated-stack evidence, not a claim that lower slices already expose public HTTP routes.

  • Locked Cargo metadata passed for the root, Python-binding and KVBM-binding workspaces; published decision-core tests: 40 passed.
  • Fresh Python binding build passed. Ten CPU HTTP/helper tests and ten manual-marker policy tests passed; malformed content types and empty questions are rejected before worker inference.
  • Nine wire/official synchronous SDK checks passed against real cached Qwen/Qwen3.8-27B revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, SGLang 0.5.19, and the final combined source 2f90adc5360cbdd5de6e5d78e9fb150cb5513595; independent-native parity was deselected.
  • Rust formatting and whitespace checks passed. A broader local Rust test rebuild hit disk exhaustion and is not counted as a passing test run; focused reruns and actual current-head remote CI remain separately tracked.

Only OpenAI /v1/decisions and Jev /v1/systemone are public decision contracts. No native SGLang wire format or nvext.format selector is supported. No new claim of independent numerical parity, Decision Index quality, full client-language coverage or physical GPU settlement is made.

Historical validation before the registry integration

The following records concern earlier source revisions and are not current-head CI evidence.

  • Final restacked tip 4fc7c6c79b0fa40e67fbf6710c0b83c5b3ad7cc1 rebuilt the Python extension with locked public dependencies and no local override; all ten HTTP integration tests and four official SDK/wire tests passed against that final extension on CPU services. Its runtime/fixture sources match the live-GPU-tested source.
  • The updated frontend stack passed 199 protocol/core tests, strict all-target Clippy and formatting; its published final tree matches the source used for the live SDK run. Dynamo pins the public frontend revision e2805e94675457ff445c261b2d12eed0d227a3b0, without machine-local dependency overrides.
  • Integrated Phase 1 source passed all ten CPU HTTP fixtures, 25 SGLang registration tests, and official-client/wire checks against the CPU mocker. Negative fixtures cover both routes and verify rejected requests do not reach workers. These are integrated-source results, not claims that every lower slice exposes HTTP.
  • Live Qwen/Qwen3.8-27B revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, SGLang 0.5.19: four wire/official synchronous Python SDK tests passed using OpenAI 3.26.0 and TypeSafe SDK 0.7.2. Same-worker native scoring comparison also passed. The independent-native-server fixture was deselected and was not requalified in this run.
  • Live GPU validation used the Phase 1 source before restacking. Final committed source equivalence and public-dependency build/CPU checks are recorded separately; historical three-contract runs do not qualify this revised scope.
  • These results do not complete async/JavaScript SDK coverage, Decision Index quality/coverage, performance/scaling, authenticated-gateway qualification, or physical GPU settlement. New-head CI is separate and is not claimed passing here.

Historical validation before the Phase 1 scope update

The following records describe earlier revisions, not current-head CI or a new qualification claim.

  • Pinned Python formatting/import checks and Ruff: passed.
  • Both qualification modules without opt-in: eight expected skips; no GPU workload launched.
  • Final Rust source passed all-target strict Clippy in the preceding schema slice; this slice changes only qualification fixtures.
  • Final-stack extension rebuild with locked public Git dependencies passed; all ten CPU endpoint tests and all 18 SGLang registration tests passed against that rebuilt environment. This is CPU/integration evidence, not a GPU rerun.

Earlier restricted-profile validation passed all three HTTP contracts and the two official synchronous Python SDK fixtures on Qwen3.8-27B/SGLang 0.5.19; independent native parity used a small Qwen model. Those historical runs are not a new GPU run of this exact restacked commit. The complete official-client matrix, including Python async and JavaScript where required, remains unfinished.

The pinned public jev-decision-index text cohort, quality/coverage reporting, performance/scaling campaigns, and physical GPU completion/abort acknowledgement remain separate release gates. No hosted-model equivalence or general authenticated-gateway qualification is claimed.

Implementation tracking: Part of #15961. This is a non-closing link to the decision-model delivery roadmap; the umbrella DEP remains open.

Review revision: explicit qualification collection

The repository-wide manual lifecycle marker and its shared marker-policy changes have been removed. The two opt-in qualification modules use the existing post_merge lifecycle only when explicitly provisioned: DYNAMO_DECISION_TEST_URL enables external-client collection, and DYNAMO_DECISION_RUN_PINNED_SGLANG=1 enables pinned-engine collection. A narrowly scoped collection hook deselects those exact modules otherwise, including explicit file invocation; ordinary CI no longer selects a profile that only skips for missing provisioning. Other serve tests remain unchanged.

This is not a new nightly lane or automated SGLang qualification claim. The qualified engine gate accepts 0.5.19 and 0.5.22. Recorded stock 0.5.22 GPU evidence is linked above at its exact source revision; rebasing or passing CPU mocker tests does not create new GPU qualification evidence. Existing selected CPU mocker coverage remains the automatic wire-contract coverage.

Validation: 21 CPU marker/selection tests passed. Actual qualification modules deselect all seven tests by default and collect all seven with explicit opt-ins, without launching GPUs during these collection checks. Scoped formatting/pre-commit and independent review passed. Earlier validation sections describe their recorded source revisions, not new-head CI or a new GPU run.

Current scope: router error handling split out

Follow-up: #17621, based on #15857 (+100 / -15 lines across seven files).

Head: 4b23c14b8f42d7fcb86c27bb03fee9744d3a4d11. The core stack remains main → #15853 → #15854 → #15855 → #15857. All lib/llm/src/kv_router* changes and their decision-side pinned-target error mapping/regression coverage are moved to a separate follow-up based on #15857. The first branch still uses the configured router (including KV-aware routing when enabled); sibling worker/rank pinning and request-local cache isolation are unchanged. Until the follow-up lands, the narrow vanished-target dispatch race retains the existing router error classification rather than the new HTTP 503 remapping.

The reduced #15853 passed 51 targeted System One tests. Scoped pre-commit passed. Earlier live-model/SDK results below describe the pre-split combined tree; the follow-up restores that tree byte-for-byte, but those runs are not new validation of the reduced core alone.

Historical main rebase and local validation

Rebased onto main 4d407c8420cbbed33be237d2582345102753d191. Prerequisites #15849–#15852 are merged. Remaining order: main → #15853 → #15854 → #15855 → #15857; each child includes its immediate parent's current head. This PR head is 84b82f444b66881186a0ccc8995d55e45430b60c, targeting bis/decisions-d6b-observability. All rewritten commits are signed; recovery refs retain the prior heads.

Integrated qualification at stack tip 84b82f444b66881186a0ccc8995d55e45430b60c passed locally with real Qwen/Qwen3-0.6B weights at revision c1899de289a04d12100db370d81485cdf75e47ca, SGLang 0.5.22, and an RTX PRO 6000 Blackwell:

  • Independent native zero-decode score parity and scoring/chat coexistence: both committed GPU tests passed.
  • Both raw public contracts plus official OpenAI 3.26.0 and TypeSafe 0.7.2 evaluation: four checks passed; oversized candidate input was rejected before inference. The SDK lane's native case was deselected because the separate GPU campaign covers it.
  • Rust selections passed: 59 decision, 11 lifecycle, 264 discovery, 73 routing-host, 14 OpenAPI, and 68 metric tests. Selections can overlap and are not summed. Strict Clippy, formatting, and pre-commit passed.

The fresh extension SHA-256 is 24f1ae18af9f57f18f720bb639c5458d717474a58fcccaa4da48e1babf8e18ac. Native parity ran before the final XPU-container-only main refresh; complete Rust/Python/test trees and root dependency manifests/lockfile are verified byte-identical after it. Official SDK checks ran after the refresh. These are integrated-stack results, not a claim that every lower slice independently exposes HTTP. No new large-model, quality, performance/scaling, authenticated-gateway, XPU-video, or physical GPU-settlement qualification is claimed. New-head GitHub CI is separate from local results.

Earlier head-specific validation below remains historical.

@devin-ai-integration

devin-ai-integration Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

✅ Dynamo PR CI passed — run 40997810781 (attempt 1) on 0b1f06fd43

Gate checks: ✅ backend-status-check · ✅ deploy-status-check · ✅ dynamo-status-check

Posted automatically by Devin for run 40997810781. Updated on every full-CI run of this PR.

@biswapanda
biswapanda added this pull request to stack #15885 October 8, 2026 06:23
@biswapanda
biswapanda marked this pull request as ready for review October 8, 2026 06:26
@biswapanda
biswapanda requested a review from a team as a code owner October 8, 2026 06:26

Copy link
Copy Markdown
Contributor Author

/ok to test 94bb237

@coderabbitai

coderabbitai Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

No actionable comments were generated in the recent review.

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: e682ab49-b3a9-4cd9-9c29-da2d636866d2
📥 Commits

Reviewing files that changed from the base of the PR and between e5fe37e and 94bb237.

📒 Files selected for processing (2)
  • tests/serve/test_decisions_external.py
  • tests/serve/test_systemone_sglang.py

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 4 remain after this review.

Walkthrough

The pull request adds opt-in tests for external decision deployments and pinned SGLang servers. The tests check wire responses, SDK success and error handling, numerical parity with native SGLang, and overlapping scoring and chat requests.

Changes

Decision-serving integration qualification

Layer / File(s) Summary
External deployment setup and wire checks
tests/serve/test_decisions_external.py
Adds configuration checks for an external decision server and response checks for OAI, SGLang-native, and SystemOne routes.
External SDK and native parity checks
tests/serve/test_decisions_external.py
Adds OpenAI and TypeSafe SDK success and error checks, plus optional numerical parity checks against a separate native SGLang server.
Pinned SGLang servers and reference scoring
tests/serve/test_systemone_sglang.py
Adds opt-in SystemOne and native SGLang server fixtures. Computes normalized label probabilities from native token log-probabilities.
SGLang parity and overlapping requests
tests/serve/test_systemone_sglang.py
Compares SystemOne and decision API results with native SGLang scores. Tests interleaved scoring and chat requests across eight workers.

Priority: Low

Estimated code review effort: 3 (Moderate) | ~25 minutes

Merge Risk: Minimal · up to 94bb2

The qualification tests have no identified merge-blocking defect. The opt-in deployment and GPU checks remain necessary to establish their intended coverage.

Pre-merge checks: Passed 4
Check name Status Explanation
Linked Issues check Passed The description references the umbrella issue #15843, dependency issue #15960, and the related PR stack. These references provide traceability for the pull request objectives.
Out of Scope Changes check Passed The changes are limited to qualification tests in tests/serve and directly support the stated native parity, wire-contract, SDK, and pinned-engine objectives. No unrelated changes are reported.
Title check Passed The title clearly summarizes the main changes: explicit native-SGLang and SDK qualification profiles.
Description check Passed The PR description is detailed and largely follows the repository template. It has a Summary, a stack and scope section, Reproduction steps, and an extensive Validation history, plus a non-closing lin…
Full details: Description check

Explanation

The PR description is detailed and largely follows the repository template. It has a Summary, a stack and scope section, Reproduction steps, and an extensive Validation history, plus a non-closing link to #15961. The template's Overview, Key Changes, Validation, and Issues and Ongoing Work headings are not used by name. The template's 'Closes #XXXX' line is absent, and the checklist item 'I used Dynamo's /pr-checklist skill' is not visibly checked. The description is long and mixes current-scope claims with superseded historical validation, which makes it hard to review. These gaps are non-critical, so the check passes.

✨ Finishing Touches: 💡 1
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Comment thread tests/serve/test_decisions_external.py Resolved
Comment thread tests/serve/test_systemone_sglang.py Resolved

@jh-nv jh-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One question about the new marker below.

Comment thread pyproject.toml Resolved
@biswapanda
biswapanda force-pushed the bis/decisions-d7-qualification branch from 2843fc6 to e001cda Compare October 28, 2026 00:54
@biswapanda
biswapanda force-pushed the bis/decisions-d7-qualification branch 2 times, most recently from 6a9f8fd to 84b82f4 Compare October 29, 2026 21:04
@biswapanda
biswapanda force-pushed the bis/decisions-d7-qualification branch from 84b82f4 to 4b23c14 Compare October 29, 2026 21:45
@biswapanda
biswapanda force-pushed the bis/decisions-d7-qualification branch 2 times, most recently from 0650f5c to ed1d185 Compare October 29, 2026 23:53
Base automatically changed from bis/decisions-d6b-observability to main October 30, 2026 03:30
@biswapanda
biswapanda requested a review from a team as a code owner October 30, 2026 03:30
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
@biswapanda
biswapanda force-pushed the bis/decisions-d7-qualification branch from ed1d185 to 0b1f06f Compare October 30, 2026 03:30
@biswapanda
biswapanda merged commit e911740 into main Oct 30, 2026
118 checks passed
@biswapanda
biswapanda deleted the bis/decisions-d7-qualification branch October 30, 2026 05:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/XXL test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants