Repository navigation
feat(mocker): default G1 to native and preserve KV event order - #12222
Conversation
WalkthroughChangesG1 backend behavior
Estimated code review effort: 3 (Moderate) | ~20 minutes 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Comment |
This comment has been minimized.
This comment has been minimized.
3cf37be to
f29a60d
Compare
779ae9e to
5ca4d4e
Compare
|
/ok to test 5ca4d4e |
PeaBrane
left a comment
There was a problem hiding this comment.
High-level follow-up: could we also hook this ordering regression into the lib/bench indexer parity coverage? The direct VllmKvManager regression is good, but this is ultimately a producer→indexer contract. It should be straightforward to run native G1 artifacts through the existing parity path and assert that the emitted Stored chain is accepted without parent-not-found errors.
jthomson04
left a comment
There was a problem hiding this comment.
Overview
Three related changes to the mocker:
- Default flip —
G1Backend::Nativeis now the enum default;MockEngineArgs::g1_backendbecomesOption<G1Backend>(tri-state: unset / explicit native / explicit kvbm). - Auto-selection replaces validation — two hard validation errors (
native + sglang,native + G2/G3/G4 offload) are deleted. Offload now resolves to KVBM innormalized(); SGLang ignores the setting entirely. - KV event ordering fix — in
G1Manager::finalize_computed_prefix, the completed-prefix publish now runs before the tail promotion.
What holds up
I traced the ordering fix against VllmKvManager::finalize_computed_prefix (vllm_backend.rs:221) and ActiveSequence::promote_computed_tail (sequence.rs:92). It's correct: with block_size=4 and computed_after=12, the finalize loop covers blocks[0..3] and breaks at the OwnedBlock::Partial tail, publishing blocks 0–1; the subsequent Promote then publishes block 2 with parent_hash = blocks[1].hash. Previously the promote fired first, so the indexer could observe a child before its parent was stored. The new regression test asserts exactly that.
Two ordering details that are right and easy to get wrong:
resolve_g1_backend()runs aftermaterialize_defaults(), so--num-g2-blocks 0(documented as "set to 0 to disable") collapses toNonefirst and correctly yields native rather than kvbm.- Resolution is idempotent, so the double-normalize path (
MockEngineArgs::default()→VllmCore::new_internalatcore.rs:484) doesn't emit the override log twice.
I also confirmed all four construction paths normalize before anyone reads the field (Default, PyO3 new, TryFrom<MockEngineArgsSerde>, VllmCore::new_internal), and that with_overrides clones-and-mutates so g1_backend survives.
Issues
1. resolved_g1_backend() can disagree with resolve_g1_backend() — lib/mocker/src/common/protocols.rs:1446
unwrap_or_default() returns Native unconditionally. On un-normalized args carrying num_g2_blocks, it returns Native + KVBM offload — precisely the combination that was a hard validation error before this PR. It's latent today because every call site normalizes first, but the rule is now encoded twice and only one copy knows about offload. Extract the predicate so they can't drift:
fn requires_kvbm_g1(&self) -> bool {
matches!(self.engine_type, EngineType::Vllm | EngineType::Trtllm)
&& (self.num_g2_blocks.is_some_and(|n| n > 0)
|| self.num_g3_blocks.is_some_and(|n| n > 0)
|| self.enable_g4_storage)
}
pub fn resolved_g1_backend(&self) -> G1Backend {
match (self.g1_backend, self.requires_kvbm_g1()) {
(_, true) => G1Backend::Kvbm,
(Some(backend), false) => backend,
(None, false) => G1Backend::Native,
}
}The is_some_and(|n| n > 0) also removes resolve_g1_backend's current unstated dependency on materialize_defaults() having already run.
2. Explicit --g1-backend native is silently overridden — lib/mocker/src/common/protocols.rs:1423
tracing::info! for discarding an explicit user selection is too quiet; warn! is the floor. Better: keep fail-fast for the explicit conflict and auto-resolve only when unset — that's the combination that preserves the old contract while getting the new ergonomics. Same gap for SGLang, where explicit native now produces no diagnostic at all; argparse-level rejection is the cheapest place to catch both.
3. Test coverage drifts off the KVBM path
The default flip silently moves ~50 tests in scheduler/vllm/tests.rs, 15 in policy/tests.rs, and most of the replay suites from KVBM to Native. Only test_running_request_catches_up_decode_tail_before_promote was pinned. KVBM is still the production backend for every offload configuration, so it shouldn't lose scheduler-level coverage. aligned_repeat_physical_occupancy_matches_vllm_and_exposes_kvbm_gap (lib/mocker/src/scheduler/vllm/policy/tests.rs:254) already has the right parametrized shape — worth applying to a representative subset.
4. The agg.rs rewrite is broader than the semantic change requires — lib/mocker/src/replay/offline/agg.rs:2059
Four coordinated changes land in one test: token counts 64→128/192, arrival 0.1→15.0, router_queue_threshold 0.5→0.0 (agg.rs:2063), and the inverted total_cached_blocks assertion. Only the last is forced by native semantics. Overriding the shared queueing_router_config helper to a 0.0 threshold makes the router queue unconditionally — in a test named ..._tracks_queue_and_cached_dispatch, that takes the threshold out of what's being exercised. Cleaner: pin the existing test to g1_backend = Kvbm (its assertions remain valid, and that path is still live) and add the native-semantics case as a new test.
Separately, agg.rs:2127:
while runtime.now_ms < 15.0 {
assert!(runtime.advance_one_timestamp().unwrap());
}is unbounded. If a future timing-model change makes advance_one_timestamp return true without advancing the clock, this hangs CI rather than failing; and if the clock steps past 15.0, the following assert_eq!(queued.now_ms, 15.0) fails for a reason unrelated to the test's subject. An iteration cap plus a >= comparison would be sturdier.
5. Docs not updated — docs/fern/dynosim/modeling.md:32
The ### G1 Managers section describes both managers but states neither the default nor the auto-selection/override rule — exactly what this PR changes. It's in-repo and belongs in this PR.
6. Minor
- Public API break:
MockEngineArgs::g1_backendispuband changedG1Backend→Option<G1Backend>. In-repo call sites are updated; external Rust consumers ofdynamo-mockerbreak. Worth a line in the PR body. - Serde asymmetry:
Serializeis derived directly, so an un-normalized args emits"g1_backend": null, whileDeserializeroutes throughinto_non_nulland rejects explicit null. Unreachable today, but#[serde(skip_serializing_if = "Option::is_none")]closes it. - Help-string consistency:
--num-g2-blocksdocuments the KVBM auto-selection;--num-g3-blocksand--enable-g4-storagedon't. Transitively covered by theg3_requires_g2/g4_requires_g2validators, so optional.
Behavior risk
This changes router-visible KV-event timing for every existing vLLM/TRT-LLM mocker user — the agg.rs diff shows cached blocks going from > 0 to 0 at t=0, because native only publishes a block once its tokens are computed. Any benchmark, planner simulation, or SLA test with numbers pinned against prior mocker output will shift. That deserves a release note; the PR body currently frames the flip as a default change without calling out the downstream numeric impact.
Security
Nothing security-relevant — configuration plumbing and event ordering in a simulation component.
Verdict
The ordering fix is correct and well-tested, and the tri-state resolution is sound at every live call site. I'd suggest items 1–4 before merge: #1 and #2 are small and remove real footguns, #3 and #4 are about not losing coverage of a backend that's still in production use.
Reviewed against refs/pull/12222/head. I did not build or run the test suite — the validation results in the PR description are taken as given.
5ca4d4e to
eaef7cc
Compare
PeaBrane
left a comment
There was a problem hiding this comment.
My comments have already been raised by @jthomson04.
|
@PeaBrane Addressed in |
|
@jthomson04 Thanks for the thorough review. Addressed the requested follow-ups in
Validation includes fmt, the relevant Clippy checks, both default and |
|
/ok to test b9880d9 |
Signed-off-by: Yongming Ding <yongmingd@nvidia.com>
Signed-off-by: Yongming Ding <yongmingd@nvidia.com>
b9880d9 to
fce5fb9
Compare
|
/ok to test fce5fb9 |
Summary
nativewith offload is also resolved to KVBM and now emits an info log describing the override.Native KV event ordering fix
The native path previously promoted the computed tail before finalizing its completed prefix. That could emit the tail's
Storedevent before the parent block'sStoredevent. The order is now reversed, and a directVllmKvManagerregression test asserts the published parent chain for a schedule that completes multiple full blocks plus a tail.Validation
cargo fmt --all -- --checkcargo clippy -p dynamo-mocker --all-targets -- -D warningscargo test -p dynamo-mocker --lib(640 passed, 1 ignored)socket.gethostbyname(socket.gethostname()))VllmKvManagerinstances initialized)