Repository navigation
test(decisions): add explicit native and SDK qualification profiles - #15857
Merged
Merged
Conversation
This was referenced Oct 8, 2026
Contributor
✅ Dynamo PR CI passed — run 40997810781 (attempt 1) on
|
Contributor
Author
|
/ok to test 94bb237 |
Contributor
biswapanda
force-pushed
the
bis/decisions-d7-qualification
branch
from
October 20, 2026 19:58
706f180 to
d8b920e
Compare
jh-nv
reviewed
Oct 20, 2026
jh-nv
left a comment
Contributor
There was a problem hiding this comment.
One question about the new marker below.
This was referenced Oct 22, 2026
biswapanda
force-pushed
the
bis/decisions-d7-qualification
branch
from
October 28, 2026 00:54
2843fc6 to
e001cda
Compare
biswapanda
force-pushed
the
bis/decisions-d7-qualification
branch
2 times, most recently
from
October 29, 2026 21:04
6a9f8fd to
84b82f4
Compare
biswapanda
force-pushed
the
bis/decisions-d7-qualification
branch
from
October 29, 2026 21:45
84b82f4 to
4b23c14
Compare
biswapanda
force-pushed
the
bis/decisions-d7-qualification
branch
2 times, most recently
from
October 29, 2026 23:53
0650f5c to
ed1d185
Compare
This was referenced Oct 30, 2026
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
biswapanda
force-pushed
the
bis/decisions-d7-qualification
branch
from
October 30, 2026 03:30
ed1d185 to
0b1f06f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 1 scope
The integrated stack supports only OpenAI
POST /v1/decisionsand Jev/TypeSafePOST /v1/systemone. Every top-levelnvextenvelope is rejected, including{}and explicit"oai". Public SGLang-native Decisions request/response types, adapters, projection and selector support are removed. SGLang/generatescoring remains an internal execution primitive, not a third public contract. Optional native-format support is deferred to DEP #15960. This scope supersedes the earlier three-contract implementation description.Stack
Dynamo review order: D1: mocker → D2: native scoring → D3: discovery → D4: admission → D5: execution → D6a: HTTP → D6b: OpenAPI → D7: qualification. Each PR targets its predecessor; merge from the base upward and update dependent bases as prerequisites land.
D7, the final Dynamo qualification-fixture slice. Base:
bis/decisions-d6b-observability.Related: umbrella DEP #15843 and its dependency plan.
Reproduction
Run within the repository's installed Dynamo test environment, with an explicitly provisioned trusted/local server.
DYNAMO_DECISION_TEST_URLis the origin without/v1, and the model value must match its registered name. This test entry point does not provision GPUs or download the large model.DYNAMO_DECISION_TEST_URL=http://localhost:8000 \ DYNAMO_DECISION_TEST_MODEL=Qwen/Qwen3.8-27B \ DYNAMO_DECISION_REQUIRE_SDKS=1 \ python -m pytest -q tests/serve/test_decisions_external.py \ -k 'not native_sglang_numeric_parity' \ --junitxml=decision-sdk-results.xmlThe previously qualified optional clients are
openai==3.26.0andtypesafe-sdk==0.7.2; install them in a dedicated qualification environment, not by changing the shared test dependency pin. For independent native parity, also set a distinctDYNAMO_DECISION_NATIVE_URLandDYNAMO_DECISION_TOKENIZER_PATHto the exact pinned local tokenizer. Raw HTTP/parity calls currently target trusted/local unauthenticated endpoints; SDK key configuration alone does not qualify an authenticated gateway.The small-model launch fixture additionally requires
DYNAMO_DECISION_RUN_PINNED_SGLANG=1and a qualified SGLang 0.5.19 or 0.5.22 runtime. Without explicit opt-in, collection deselects it rather than launching a GPU workload. This does not qualify arbitrary newer runtimes or configurations.Validation
The stack is rebased onto main
532616d12ddea4d9ebce30fbcb67807cc02b63e0, with every remaining PR based on its updated predecessor. It preserves main's releaseddynamo-protocols =8.4.0anddynamo-parsers =10.2.2alongsidedynamo-decisions =0.1.0. Validation below is historical evidence at the source revisions explicitly identified; the review-update comment records final-head validation.Recorded integrated-stack validation and complete GPU pytest output ran at tip
876e2980df44ac3ec16d3079965e1c58039e5cd3: both committed native SGLang tests passed on stock 0.5.22/GB200 with pinned Qwen3-0.6B; four official OpenAI/TypeSafe client/wire cases passed separately against the live worker. The incompatible 43-choice Jev probe returned the new actionable HTTP 422 diagnostic. These are integrated-stack results, not a claim that lower slices expose public routes independently.Prior recorded source validation also passed 294 mocker tests, 51 System One tests, 52 frontend service tests, nine live CPU HTTP tests, focused routing/protocol/lifecycle/OpenAPI and Python regressions, strict LLM all-target Clippy without default features, formatting/pre-commit, and locked metadata. Documentation configuration and broken-link checks passed. Counts overlap; new-head remote CI is tracked separately. This does not qualify the 27B model, model quality, performance/scaling, or physical GPU cancellation settlement.
Historical validation before this main rebase
Rebased onto main
02da7f44861b65758b960770e0763e9faa8e4ca4. D3 and later use crates.iodynamo-decisions =0.1.0withdynamo-protocols =8.2.0; obsolete Git dependency/source patches are removed. The prior unpublished-crate prerequisite is no longer applicable.Validation below is integrated-stack evidence, not a claim that lower slices already expose public HTTP routes.
1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, SGLang 0.5.19, and the final combined source2f90adc5360cbdd5de6e5d78e9fb150cb5513595; independent-native parity was deselected.Only OpenAI
/v1/decisionsand Jev/v1/systemoneare public decision contracts. No native SGLang wire format ornvext.formatselector is supported. No new claim of independent numerical parity, Decision Index quality, full client-language coverage or physical GPU settlement is made.Historical validation before the registry integration
The following records concern earlier source revisions and are not current-head CI evidence.
4fc7c6c79b0fa40e67fbf6710c0b83c5b3ad7cc1rebuilt the Python extension with locked public dependencies and no local override; all ten HTTP integration tests and four official SDK/wire tests passed against that final extension on CPU services. Its runtime/fixture sources match the live-GPU-tested source.e2805e94675457ff445c261b2d12eed0d227a3b0, without machine-local dependency overrides.Qwen/Qwen3.8-27Brevision1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, SGLang 0.5.19: four wire/official synchronous Python SDK tests passed using OpenAI 3.26.0 and TypeSafe SDK 0.7.2. Same-worker native scoring comparison also passed. The independent-native-server fixture was deselected and was not requalified in this run.Historical validation before the Phase 1 scope update
The following records describe earlier revisions, not current-head CI or a new qualification claim.
Earlier restricted-profile validation passed all three HTTP contracts and the two official synchronous Python SDK fixtures on Qwen3.8-27B/SGLang 0.5.19; independent native parity used a small Qwen model. Those historical runs are not a new GPU run of this exact restacked commit. The complete official-client matrix, including Python async and JavaScript where required, remains unfinished.
The pinned public
jev-decision-indextext cohort, quality/coverage reporting, performance/scaling campaigns, and physical GPU completion/abort acknowledgement remain separate release gates. No hosted-model equivalence or general authenticated-gateway qualification is claimed.Implementation tracking: Part of #15961. This is a non-closing link to the decision-model delivery roadmap; the umbrella DEP remains open.
Review revision: explicit qualification collection
The repository-wide
manuallifecycle marker and its shared marker-policy changes have been removed. The two opt-in qualification modules use the existingpost_mergelifecycle only when explicitly provisioned:DYNAMO_DECISION_TEST_URLenables external-client collection, andDYNAMO_DECISION_RUN_PINNED_SGLANG=1enables pinned-engine collection. A narrowly scoped collection hook deselects those exact modules otherwise, including explicit file invocation; ordinary CI no longer selects a profile that only skips for missing provisioning. Other serve tests remain unchanged.This is not a new nightly lane or automated SGLang qualification claim. The qualified engine gate accepts 0.5.19 and 0.5.22. Recorded stock 0.5.22 GPU evidence is linked above at its exact source revision; rebasing or passing CPU mocker tests does not create new GPU qualification evidence. Existing selected CPU mocker coverage remains the automatic wire-contract coverage.
Validation: 21 CPU marker/selection tests passed. Actual qualification modules deselect all seven tests by default and collect all seven with explicit opt-ins, without launching GPUs during these collection checks. Scoped formatting/pre-commit and independent review passed. Earlier validation sections describe their recorded source revisions, not new-head CI or a new GPU run.
Current scope: router error handling split out
Follow-up: #17621, based on #15857 (+100 / -15 lines across seven files).
Head:
4b23c14b8f42d7fcb86c27bb03fee9744d3a4d11. The core stack remains main → #15853 → #15854 → #15855 → #15857. Alllib/llm/src/kv_router*changes and their decision-side pinned-target error mapping/regression coverage are moved to a separate follow-up based on #15857. The first branch still uses the configured router (including KV-aware routing when enabled); sibling worker/rank pinning and request-local cache isolation are unchanged. Until the follow-up lands, the narrow vanished-target dispatch race retains the existing router error classification rather than the new HTTP 503 remapping.The reduced #15853 passed 51 targeted System One tests. Scoped pre-commit passed. Earlier live-model/SDK results below describe the pre-split combined tree; the follow-up restores that tree byte-for-byte, but those runs are not new validation of the reduced core alone.
Historical main rebase and local validation
Rebased onto main
4d407c8420cbbed33be237d2582345102753d191. Prerequisites #15849–#15852 are merged. Remaining order: main → #15853 → #15854 → #15855 → #15857; each child includes its immediate parent's current head. This PR head is84b82f444b66881186a0ccc8995d55e45430b60c, targetingbis/decisions-d6b-observability. All rewritten commits are signed; recovery refs retain the prior heads.Integrated qualification at stack tip
84b82f444b66881186a0ccc8995d55e45430b60cpassed locally with real Qwen/Qwen3-0.6B weights at revisionc1899de289a04d12100db370d81485cdf75e47ca, SGLang 0.5.22, and an RTX PRO 6000 Blackwell:The fresh extension SHA-256 is
24f1ae18af9f57f18f720bb639c5458d717474a58fcccaa4da48e1babf8e18ac. Native parity ran before the final XPU-container-only main refresh; complete Rust/Python/test trees and root dependency manifests/lockfile are verified byte-identical after it. Official SDK checks ran after the refresh. These are integrated-stack results, not a claim that every lower slice independently exposes HTTP. No new large-model, quality, performance/scaling, authenticated-gateway, XPU-video, or physical GPU-settlement qualification is claimed. New-head GitHub CI is separate from local results.Earlier head-specific validation below remains historical.