Skip to content

feat(mocker): default G1 to native and preserve KV event order - #12222

Merged
dreamtalen merged 2 commits into
mainfrom
codex/mocker-native-default
Jul 28, 2026
Merged

dreamtalen merged 2 commits into
mainfrom
codex/mocker-native-default

Conversation

@dreamtalen

@dreamtalen dreamtalen commented Jul 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Make native G1 the automatic default for vLLM and TensorRT-LLM mock schedulers; SGLang continues to use its own KV manager.
  • Treat G1 backend selection as tri-state: an omitted setting resolves to native unless G2/G3/G4 offload requires KVBM. Explicit native with offload is also resolved to KVBM and now emits an info log describing the override.
  • Fix native G1 event ordering by publishing completed parent blocks before promoting and publishing the mutable tail. This prevents the router indexer from observing a child whose parent has not been stored yet.
  • Keep compatibility-only KVBM tests explicit and update offline replay expectations for computed-only native cache visibility.

Native KV event ordering fix

The native path previously promoted the computed tail before finalizing its completed prefix. That could emit the tail's Stored event before the parent block's Stored event. The order is now reversed, and a direct VllmKvManager regression test asserts the published parent chain for a schedule that completes multiple full blocks plus a tail.

Validation

  • cargo fmt --all -- --check
  • cargo clippy -p dynamo-mocker --all-targets -- -D warnings
  • cargo test -p dynamo-mocker --lib (640 passed, 1 ignored)
  • Focused Python G1 configuration tests (11 passed)
  • Python configuration suite (48 passed; one unrelated local hostname DNS failure in socket.gethostbyname(socket.gethostname()))
  • Offline KV-router replay (16/16 requests; two VllmKvManager instances initialized)
  • Direct PyO3 selection check, including the explicit-native override log
  • Relevant pre-commit hooks passed; repository-wide pytest marker collection was blocked by an unrelated local port allocation failure

@copy-pr-bot

copy-pr-bot Bot commented Jul 27, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@dreamtalen
dreamtalen marked this pull request as ready for review July 27, 2026 21:07
@dreamtalen
dreamtalen requested a review from a team as a code owner July 27, 2026 21:07
@github-actions github-actions Bot added the fix label Jul 27, 2026
@coderabbitai

coderabbitai Bot commented Jul 27, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Walkthrough

Changes

G1 backend behavior

Layer / File(s) Summary
Backend resolution and CLI defaults
components/src/dynamo/mocker/args.py, components/src/dynamo/mocker/config.py, components/src/dynamo/mocker/tests/unit/test_config.py
CLI and JSON engine arguments now default to native G1, select KVBM for applicable offload storage settings, and test engine-specific and explicit-backend behavior.
SGLang validation behavior
lib/mocker/src/common/protocols.rs
SGLang accepts either G1 backend value without triggering the native-G1 validation error, while legacy offload conflicts remain validated for other engines and are covered by tests.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description has useful detail, but it does not follow the required template and is missing the reviewer-start and related-issues sections. Reformat to the repo template and add Overview, Details, Where should the reviewer start?, and a required Related Issues section with an issue link or confirmed no issue.
✅ Passed checks (4 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title matches the main change: defaulting mocker G1 to native, with an extra note covering the event-ordering fix.

Comment @coderabbitai help to get the list of available commands.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

Comment thread components/src/dynamo/mocker/config.py Resolved
@dreamtalen
dreamtalen marked this pull request as draft July 27, 2026 21:25
Comment thread components/src/dynamo/mocker/config.py Resolved
@datadog-official

This comment has been minimized.

@dreamtalen
dreamtalen force-pushed the codex/mocker-native-default branch from 3cf37be to f29a60d Compare July 27, 2026 21:56
@dreamtalen dreamtalen changed the title fix(mocker): default G1 backend to native feat(mocker): default G1 backend to native Jul 27, 2026
@github-actions github-actions Bot added feat and removed fix labels Jul 27, 2026
@dreamtalen
dreamtalen force-pushed the codex/mocker-native-default branch 2 times, most recently from 779ae9e to 5ca4d4e Compare July 27, 2026 23:30
@dreamtalen dreamtalen changed the title feat(mocker): default G1 backend to native feat(mocker): default G1 to native and preserve KV event order Jul 27, 2026
@dreamtalen
dreamtalen marked this pull request as ready for review July 27, 2026 23:45
@dreamtalen
dreamtalen requested review from a team as code owners July 27, 2026 23:45
@dreamtalen

Copy link
Copy Markdown
Contributor Author

/ok to test 5ca4d4e

@PeaBrane PeaBrane left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

High-level follow-up: could we also hook this ordering regression into the lib/bench indexer parity coverage? The direct VllmKvManager regression is good, but this is ultimately a producer→indexer contract. It should be straightforward to run native G1 artifacts through the existing parity path and assert that the emitted Stored chain is accepted without parent-not-found errors.

Comment thread lib/bindings/python/rust/llm/replay.rs Resolved

@jthomson04 jthomson04 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overview

Three related changes to the mocker:

  1. Default flip — G1Backend::Native is now the enum default; MockEngineArgs::g1_backend becomes Option<G1Backend> (tri-state: unset / explicit native / explicit kvbm).
  2. Auto-selection replaces validation — two hard validation errors (native + sglang, native + G2/G3/G4 offload) are deleted. Offload now resolves to KVBM in normalized(); SGLang ignores the setting entirely.
  3. KV event ordering fix — in G1Manager::finalize_computed_prefix, the completed-prefix publish now runs before the tail promotion.

What holds up

I traced the ordering fix against VllmKvManager::finalize_computed_prefix (vllm_backend.rs:221) and ActiveSequence::promote_computed_tail (sequence.rs:92). It's correct: with block_size=4 and computed_after=12, the finalize loop covers blocks[0..3] and breaks at the OwnedBlock::Partial tail, publishing blocks 0–1; the subsequent Promote then publishes block 2 with parent_hash = blocks[1].hash. Previously the promote fired first, so the indexer could observe a child before its parent was stored. The new regression test asserts exactly that.

Two ordering details that are right and easy to get wrong:

  • resolve_g1_backend() runs after materialize_defaults(), so --num-g2-blocks 0 (documented as "set to 0 to disable") collapses to None first and correctly yields native rather than kvbm.
  • Resolution is idempotent, so the double-normalize path (MockEngineArgs::default() → VllmCore::new_internal at core.rs:484) doesn't emit the override log twice.

I also confirmed all four construction paths normalize before anyone reads the field (Default, PyO3 new, TryFrom<MockEngineArgsSerde>, VllmCore::new_internal), and that with_overrides clones-and-mutates so g1_backend survives.

Issues

1. resolved_g1_backend() can disagree with resolve_g1_backend() — lib/mocker/src/common/protocols.rs:1446

unwrap_or_default() returns Native unconditionally. On un-normalized args carrying num_g2_blocks, it returns Native + KVBM offload — precisely the combination that was a hard validation error before this PR. It's latent today because every call site normalizes first, but the rule is now encoded twice and only one copy knows about offload. Extract the predicate so they can't drift:

fn requires_kvbm_g1(&self) -> bool {
    matches!(self.engine_type, EngineType::Vllm | EngineType::Trtllm)
        && (self.num_g2_blocks.is_some_and(|n| n > 0)
            || self.num_g3_blocks.is_some_and(|n| n > 0)
            || self.enable_g4_storage)
}

pub fn resolved_g1_backend(&self) -> G1Backend {
    match (self.g1_backend, self.requires_kvbm_g1()) {
        (_, true) => G1Backend::Kvbm,
        (Some(backend), false) => backend,
        (None, false) => G1Backend::Native,
    }
}

The is_some_and(|n| n > 0) also removes resolve_g1_backend's current unstated dependency on materialize_defaults() having already run.

2. Explicit --g1-backend native is silently overridden — lib/mocker/src/common/protocols.rs:1423

tracing::info! for discarding an explicit user selection is too quiet; warn! is the floor. Better: keep fail-fast for the explicit conflict and auto-resolve only when unset — that's the combination that preserves the old contract while getting the new ergonomics. Same gap for SGLang, where explicit native now produces no diagnostic at all; argparse-level rejection is the cheapest place to catch both.

3. Test coverage drifts off the KVBM path

The default flip silently moves ~50 tests in scheduler/vllm/tests.rs, 15 in policy/tests.rs, and most of the replay suites from KVBM to Native. Only test_running_request_catches_up_decode_tail_before_promote was pinned. KVBM is still the production backend for every offload configuration, so it shouldn't lose scheduler-level coverage. aligned_repeat_physical_occupancy_matches_vllm_and_exposes_kvbm_gap (lib/mocker/src/scheduler/vllm/policy/tests.rs:254) already has the right parametrized shape — worth applying to a representative subset.

4. The agg.rs rewrite is broader than the semantic change requires — lib/mocker/src/replay/offline/agg.rs:2059

Four coordinated changes land in one test: token counts 64→128/192, arrival 0.1→15.0, router_queue_threshold 0.5→0.0 (agg.rs:2063), and the inverted total_cached_blocks assertion. Only the last is forced by native semantics. Overriding the shared queueing_router_config helper to a 0.0 threshold makes the router queue unconditionally — in a test named ..._tracks_queue_and_cached_dispatch, that takes the threshold out of what's being exercised. Cleaner: pin the existing test to g1_backend = Kvbm (its assertions remain valid, and that path is still live) and add the native-semantics case as a new test.

Separately, agg.rs:2127:

while runtime.now_ms < 15.0 {
    assert!(runtime.advance_one_timestamp().unwrap());
}

is unbounded. If a future timing-model change makes advance_one_timestamp return true without advancing the clock, this hangs CI rather than failing; and if the clock steps past 15.0, the following assert_eq!(queued.now_ms, 15.0) fails for a reason unrelated to the test's subject. An iteration cap plus a >= comparison would be sturdier.

5. Docs not updated — docs/fern/dynosim/modeling.md:32

The ### G1 Managers section describes both managers but states neither the default nor the auto-selection/override rule — exactly what this PR changes. It's in-repo and belongs in this PR.

6. Minor

  • Public API break: MockEngineArgs::g1_backend is pub and changed G1Backend → Option<G1Backend>. In-repo call sites are updated; external Rust consumers of dynamo-mocker break. Worth a line in the PR body.
  • Serde asymmetry: Serialize is derived directly, so an un-normalized args emits "g1_backend": null, while Deserialize routes through into_non_null and rejects explicit null. Unreachable today, but #[serde(skip_serializing_if = "Option::is_none")] closes it.
  • Help-string consistency: --num-g2-blocks documents the KVBM auto-selection; --num-g3-blocks and --enable-g4-storage don't. Transitively covered by the g3_requires_g2 / g4_requires_g2 validators, so optional.

Behavior risk

This changes router-visible KV-event timing for every existing vLLM/TRT-LLM mocker user — the agg.rs diff shows cached blocks going from > 0 to 0 at t=0, because native only publishes a block once its tokens are computed. Any benchmark, planner simulation, or SLA test with numbers pinned against prior mocker output will shift. That deserves a release note; the PR body currently frames the flip as a default change without calling out the downstream numeric impact.

Security

Nothing security-relevant — configuration plumbing and event ordering in a simulation component.

Verdict

The ordering fix is correct and well-tested, and the tri-state resolution is sound at every live call site. I'd suggest items 1–4 before merge: #1 and #2 are small and remove real footguns, #3 and #4 are about not losing coverage of a backend that's still in production use.

Reviewed against refs/pull/12222/head. I did not build or run the test suite — the validation results in the PR description are taken as given.

@dreamtalen
dreamtalen force-pushed the codex/mocker-native-default branch from 5ca4d4e to eaef7cc Compare July 28, 2026 00:29

@PeaBrane PeaBrane left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My comments have already been raised by @jthomson04.

@dreamtalen
dreamtalen requested a review from a team as a code owner July 28, 2026 06:21
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

@PeaBrane Addressed in eaef7cc266. I added a Native G1 parent-chain artifact and replay it through the existing lib/bench parity path across all five indexer variants. The regression asserts zero Stored/parent_not_found events and verifies that the complete three-block chain is indexed. This complements the direct VllmKvManager test by covering the producer→indexer contract end to end. The mooncake_trace suite passes with 16 passed and 1 ignored.

Copy link
Copy Markdown
Contributor Author

@jthomson04 Thanks for the thorough review. Addressed the requested follow-ups in b9880d9b55:

  • Extracted the shared requires_kvbm_g1() predicate so resolved_g1_backend() and normalization cannot drift, including on unnormalized positive offload values.
  • Restored fail-fast behavior for explicit native + G2/G3/G4 offload on vLLM/TensorRT-LLM; only an unset backend auto-selects KVBM. SGLang intentionally ignores g1_backend because it uses its own KV manager and does not support KVBM offload.
  • Kept representative KVBM scheduler/offload coverage explicitly pinned and run the completion-event regression for both Native and KVBM.
  • Restored the existing agg.rs snapshot to its historical KVBM semantics instead of changing its timing and queue-threshold behavior.
  • Added skip_serializing_if for the optional backend, aligned the CLI help text, and documented backend selection plus the router-visible timing/numeric replay impact.

Validation includes fmt, the relevant Clippy checks, both default and kvbm-offload mocker test suites, focused Python config tests, and the mooncake_trace parity suite.

@dreamtalen

Copy link
Copy Markdown
Contributor Author

/ok to test b9880d9

@github-actions

github-actions Bot commented Jul 28, 2026 •

Copy link
Copy Markdown
Contributor

Signed-off-by: Yongming Ding <yongmingd@nvidia.com>
Signed-off-by: Yongming Ding <yongmingd@nvidia.com>
@dreamtalen
dreamtalen force-pushed the codex/mocker-native-default branch from b9880d9 to fce5fb9 Compare July 28, 2026 06:53
@dreamtalen

Copy link
Copy Markdown
Contributor Author

/ok to test fce5fb9

@dreamtalen
dreamtalen enabled auto-merge (squash) July 28, 2026 22:01
@dreamtalen
dreamtalen merged commit 6125e82 into main Jul 28, 2026
110 checks passed
@dreamtalen
dreamtalen deleted the codex/mocker-native-default branch July 28, 2026 22:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation feat size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants