Repository navigation
feat(decisions): validate native scores and answer-position tokenization - #15850
Conversation
✅ Dynamo PR CI passed — run 40454960805 (attempt 1) on
|
|
/ok to test a0b6ee5 |
a0b6ee5 to
7aa8afa
Compare
7aa8afa to
46a9813
Compare
luckyq
left a comment
There was a problem hiding this comment.
Response validation looks right: order/cardinality enforced against the request, non-null token text rejected, output_ids:[42]+completion_tokens:0 and []+1 both rejected, D1 mocker envelope compatible. Approving; please follow up on test coverage:
lib/llm/src/preprocessor.rs:3049-3135: the label-tokenization checks (encoded.len() == prompt_ids.len() + 1, prefix equality, distinct label ids) have no test. Both new tests exit before the label loop and use a byte-per-token stub, so deleting those checks passes the suite. A positive + negative case with a real tokenizer (e.g. mock-llama, including a leading-space label) would cover the BPE boundary behaviour the feature depends on.preprocessor.rs:3080: the reasoning-start check usesreasoning_parserdirectly; the chat path goes throughconfigured_family(tool_call_parser, reasoning_parser). With onlytool_call_parser=gemma4/kimi_k3/minimax_m3 set, the generic<think>check misses those families' markers.preprocessor.rs:3107: a round-trip mismatch caused by user content containing special-token text is classified asSystemOneModelIncompatible, i.e. a per-request problem reported as a model-level one.
|
fixed in 3c1269d. Review follow-up: real TinyLlama BPE fixtures now exercise leading-space labels, multi-token labels, duplicate normalized IDs, and append-only prefix mismatch. User-content round-trip failures are classified as invalid requests rather than model incompatibility. Reasoning-family detection now accounts for tool-parser configuration; 30e5e76 adds the narrowly scoped MiniMax opener follow-up without changing chat preprocessing. Targeted protocol and preprocessing regressions passed. |
30e5e76 to
5555357
Compare
5555357 to
c6ec85a
Compare
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
Signed-off-by: Biswa Panda <biswa.panda@gmail.com>
c6ec85a to
31bf442
Compare
Summary
Stack
Dynamo review order: D1: mocker → D2: native scoring → D3: discovery → D4: admission → D5: execution → D6a: HTTP → D6b: OpenAPI → D7: qualification. Each PR targets its predecessor; merge from the base upward and update dependent bases as prerequisites land.
D2 of the Dynamo decision API stack; depends on D1: mocker scoring. Base:
bis/decisions-d1-mocker.This slice introduces native transport and tokenizer primitives only. Discovery-bound preparation and branch execution follow in later slices; public routes remain absent. D2 itself does not require the new frontend crate, so it can be reviewed independently of that cross-repository release boundary.
Related: umbrella DEP #15843 and its PR dependency plan.
Validation
Current stack rebased onto main
acef8afe0a5b6c7ae9acec3d1ad4fd76234efc5f, with every PR targeting its latest predecessor. Signed rebase commits supersede the earlier published heads.Fresh integrated-stack validation and complete GPU pytest output ran at tip
876e2980df44ac3ec16d3079965e1c58039e5cd3: both committed native SGLang tests passed on stock 0.5.22/GB200 with pinned Qwen3-0.6B; four official OpenAI/TypeSafe client/wire cases passed separately against the live worker. The incompatible 43-choice Jev probe returned the new actionable HTTP 422 diagnostic. These are integrated-stack results, not a claim that lower slices expose public routes independently.Final-source validation also passed 294 mocker tests, 51 System One tests, 52 frontend service tests, nine live CPU HTTP tests, focused routing/protocol/lifecycle/OpenAPI and Python regressions, strict LLM all-target Clippy without default features, formatting/pre-commit, and locked metadata. Documentation configuration and broken-link checks passed. Counts overlap; new-head remote CI is tracked separately. This does not qualify the 27B model, model quality, performance/scaling, or physical GPU cancellation settlement.
Historical validation before this main rebase
Rebased onto main
02da7f44861b65758b960770e0763e9faa8e4ca4. D3 and later use crates.iodynamo-decisions =0.1.0withdynamo-protocols =8.2.0; obsolete Git dependency/source patches are removed. The prior unpublished-crate prerequisite is no longer applicable.Validation below is integrated-stack evidence, not a claim that lower slices already expose public HTTP routes.
1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, SGLang 0.5.19, and the final combined source2f90adc5360cbdd5de6e5d78e9fb150cb5513595; independent-native parity was deselected.Only OpenAI
/v1/decisionsand Jev/v1/systemoneare public decision contracts. No native SGLang wire format ornvext.formatselector is supported. No new claim of independent numerical parity, Decision Index quality, full client-language coverage or physical GPU settlement is made.Historical validation before the registry integration
The following records concern earlier source revisions and are not current-head CI evidence.
3213fe6f03.These tests validate transport/preprocessing contracts, not broad model quality or physical GPU cancellation settlement. GPU numerical parity and official-client fixtures are retained in the qualification slice.
Summary by CodeRabbit
Implementation tracking: Part of #15961. This is a non-closing link to the decision-model delivery roadmap; the umbrella DEP remains open.