Repository navigation
refactor(aisimulate): extract generalized mocker engine and replayer - #12525
Conversation
9164057 to
dfcce49
Compare
tedzhouhk
left a comment
There was a problem hiding this comment.
I finished reviewing the public API. The engine/replayer extraction and the Engine/Dynamo composition boundaries look good to me; I do not have API concerns with those parts.
I am requesting changes specifically for the user-facing CLI. The proposed aisimulate predict interface mixes several abstraction levels and is difficult to reason about. A few examples:
--island--oslare top-level CLI flags, whileconcurrency,request_count,request_rate, and arrival controls for the same traffic model live in YAML, sometimes at different YAML levels.- Trace input comes through a separate
--trafficYAML, whileconfig.workloadcan override it. This gives one workload multiple input sources with non-obvious precedence. - Deployment is split between CLI flags (
model,backend,system,tp-size, andreplicas) and config fields (num_workers,engine_args, andprefill/decode), with multiple aliases for some fields. concurrencyappears both insideworkloadand as a top-levelReplaySpecfield. The latter is understandable as a Sweeper-resolved value for KV-load candidates, but exposing that intermediate contract through a concrete one-shot CLI is confusing.- The existing replay CLI's domain split—traffic, Router, prefill/decode engines, and Planner—is easier for users to understand.
Please defer the CLI changes from this PR for now. I suggest landing the engine/replayer/API extraction without establishing this new public CLI contract. Keep single-run replay and Sweeper as two separate user-facing paths for the moment; do not partially combine them through aisimulate predict until we have discussed and agreed on a coherent, typed CLI/config model. We can introduce the consolidated predict/sweep surface in a follow-up after that design discussion.
tedzhouhk
left a comment
There was a problem hiding this comment.
Independent adversarial verification confirmed the two blocking P1s and the three scoped P2s below. I am not including the previously considered mid-pass CancelDestination issue because both shipped drivers defer that command until the pass has completed.
dfcce49 to
4031f29
Compare
|
@tedzhouhk Thanks, removed the new |
4031f29 to
40c1791
Compare
|
/ok to test 6a6556e |
Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
|
/ok to test 293c9c1 |
Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
|
/ok to test b1172bf |
Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
|
/ok to test 6b3d991 |
saturley-hall
left a comment
There was a problem hiding this comment.
With the passing PR workflow I think this satisfies the minimal ops requirements.
Note we will need to file a follow-up PR to enable aisimulate to go out with the nightly dynamo build and update the dependency to that nightly version so that the latest work is verifiable.
Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
|
/ok to test f1e68 |
Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
|
/ok to test 561ca5e |
Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
|
/ok to test daeb2ac |
|
Hi @dreamtalen, I've been using the kvbm offloading feature in DynoSim. Since this feature was removed in this refactor, would it be added back later? Thanks. |
|
Hi @leigao97, we plan to restore kv offload simulation in the AISimulate repo https://github.com/ai-dynamo/aisimulate, but not through KVBM, since KVBM is being sunset from Dynamo. Could you share which specific KVBM offloading features or workflows you have been using? That would help us prioritize the replacement. Thanks! |
|
Thank you, @dreamtalen, for the quick reply. I was primarily using the KVBM offloading feature to simulate three-tiered (HBM/DRAM/SSD) prefix cache offloading with configurable capacity and bandwidth, similar to the HiCache design in SGLang. Having an accurate simulation of these multi-tiered offloading behaviors in AISimulate would be incredibly valuable for my workflows. |
Summary
This PR extracts Dynamo-neutral simulation execution into AISimulate while keeping Dynamo-specific composition in Dynamo.
aisimulate-coreis the public Rust crate. Itsenginemodule owns vLLM, SGLang, and TensorRT-LLM scheduler simulation, framework-local G1 KV accounting, preemption, timing, single-rank execution, and attention-DP grouping.aisimulate-core::replayowns deterministic virtual time, logical-worker lifecycle, aggregated and disaggregated replay, placement/scaling seams, artifacts, canonical reports, and execution evidence.EngineConfig,TimingModel,TimingModelConfig,ReplaySpec,ReplayReport, andReplayerfacade. Advanced contracts remain under explicitengineandreplaypaths.aisimulate._runtimebinding expose engine-only and Dynamo-composed replay to the CLI and Sweeper boundaries.The dependency and module direction is:
aisimulate-corehas no Dynamo, KVBM, or Tokio dependency.Review surface
The raw GitHub diff substantially overstates the implementation delta because file extraction is counted on both the deleted and added sides. The following line-level analysis separates pre-existing code, mechanical edits, tests, and the extracted or reworked review surface.
Breaking changes
python -m dynamo.mockerand replay options are removed:--g1-backend,--num-g2-blocks,--num-g3-blocks,--enable-g4-storage,--offload-batch-size, and the--bandwidth-g*-to-g*-gbpsoptions. Framework-native G2 simulation is separate future work.dynamo-kv-replay-parityprotocol now covers native G1. Historical KVBM parity campaigns must use a historical revision and its matching protocol.Ownership and entrypoints
aisimulate-coreaisimulate-core::engineaisimulate-core::replayCurrent entrypoints are:
python -m aisimulate.replaypython -m dynamo.replaySweeper(runner_factory=...).run(config)and the examples underaisimulate/examples/sweeper/The two replay modules share their engine, topology, traffic, replay-mode, SLA, and output argument schema. The Dynamo entrypoint adds optional Router, Planner, AIC, and online integration; its no-adapter form remains compatible with engine-only replay.
Correctness hardening
The review and latest-main reconciliation preserve one execution path while adding focused safeguards:
29930c4ceaincorporates fix(mocker): truncate disagg prefill output plan #13052; exact output-token plans remain authoritative, and disaggregated prefill truncates its local plan without mutating decode.LiveEngine::submit()bursts are forwarded without limiting admission to one request per pass; the public compatibility request lane remains unbounded while grouped-engine command lanes remain bounded.API walkthrough
Expand API walkthrough
Stable facade
aisimulate-coreexposes the common stable surface at the crate root:Advanced engine and adapter contracts use explicit
aisimulate_core::engineandaisimulate_core::replaypaths.Overall flow
Live Mocker does not use the virtual-time Replayer, but shares its grouped engine:
Generalized Mocker Engine
Main files:
aisimulate/crates/core/src/engine/generalized/contracts.rsaisimulate/crates/core/src/engine/generalized/engine.rsaisimulate/crates/core/src/engine/scheduler/SchedulerRankowns one rank's scheduler, KV accounting, preemption, and timing state.GeneralizedMockerEngine<C>owns attention-DP membership, readiness, pass barriers, and modeled group completion. Offline replay supplies virtual time; Live Mocker supplies Tokio/wall-clock events through a Dynamo-owned driver.Replayer
Main files:
aisimulate/crates/core/src/replay/replayer.rsaisimulate/crates/core/src/replay/spec.rsaisimulate/crates/core/src/replay/engine.rsReplayer<C: ReplayComposition>owns virtual time, the event queue, logical-worker lifecycle, Agg/Disagg runtime state, trace collection, artifacts, and reports.Replayer::new(...)provides round-robin placement with no scaling.Replayer::with_composition(...)accepts Dynamo-owned placement and scaling policies.Replayer::run_with_artifacts(...)uses the shared aggregated runtime with an optional Replay-owned artifact sink.Placement, scaling, engine, and invariant failures retain typed provenance.
Python runner and CLI boundary
Main files:
aisimulate/src/aisimulate/runner.pyaisimulate/src/aisimulate/replay/components/src/dynamo/replay/simulation.pycomponents/src/dynamo/replay/main.pyEngineReplayRunnermaterializes a serializableReplaySpec, callsaisimulate._runtime, and normalizesReplayReport.DynamoReplayRunnerimplements the same Runner contract while constructing Dynamo Router and Planner providers.python -m dynamo.replaycomposes the shared AISimulate base parser/lowering with Dynamo-only extensions; it does not duplicate the engine-only schema.Router, Planner, and Live adapters
Dynamo reuses the existing
PlacementPolicy<Request>boundary, including round-robin andKvRouterPlacement, throughReplayComposition. Planner is injected through the scaling-policy seam.Live Mocker constructs the same grouped engine as offline replay. Dynamo remains responsible for Tokio driving, transport, cancellation, output publication, and production metrics.
Validation
Final HEAD:
23518163ebRust
cargo test -p aisimulate-core -p dynamo-mocker -p dynamo-mocker-backend --no-fail-fastaisimulate-core: 449 tests passeddynamo-mocker: 194 tests passeddynamo-mocker-backend: 20 tests passedcargo clippy -p aisimulate-core -p dynamo-mocker -p dynamo-mocker-backend --all-targets -- -D warningscargo fmt --all --checkpre-commit run --all-filesgit diff --checkcargo metadata --lockedchecks passed.cargo metadata --locked; both builders now copyaisimulate/crates/alongsidelib/before Cargo resolves the root workspace.cargo metadata --lockedand built the native AISimulate wheel; a regression test now enforces complete workspace COPY coverage.Focused regressions cover grouped shutdown, production-lane submit batching, attention-DP FPM normalization and rank identity, vLLM/SGLang MTP lifecycle boundaries, cached-prefix propagation, SGLang cache-metric latching, bounded same-timestamp retry, pass-end KV visibility, topology release evidence, artifact capture, unsupported topology/selector rejection, and SGLang prefix-caching behavior.
Python and packaging
linux/amd64andlinux/arm64, is readable as the CI runner identity (1001:1001), and collects the intended 25 optional AISimulate/Planner tests.ai_dynamo_runtimewheels passed the exact Planner selector (25 passed, 134 deselected), the online KV Router CLI regressions (2 passed), and SGLang AIC parity (6 passed, 6 deselected).aisimulate._runtimeon Python 3.12.ai-dynamodependency and no console script.Replay parity
The authoritative native-G1 correctness campaign compared candidate
23518163ebwith pinned merge-base main975d6a15a6using the first 5,000 requests from the canonical Mooncake trace. The frozen slice SHA-256 is3892ae19ae480b643155f0c6b9d798591cbe2e73bec6a0fa5ae3d3bc0332fb8a, matching thedynamo-kv-replay-parityprotocol.The four skill-aligned KV-aware rows used the current qualified internal-polynomial seeds. Each baseline and candidate row ran twice in a separate process; every revision/row pair produced one deterministic digest, completed 5,000/5,000 requests, and matched after accounting for the intentionally retired KVBM configuration schema:
cbda006db3b1b4c7fe4eda15412469aa237b5ada44d1aaa39c5261b136ef6180b69f998ec1978f5bf00f172d7648467af730e29cb57d7e42210664fbc220ad27dd693851bdc43cff67fe049ef77cf0f4d41e85c869a0bfd995a85d72147e6ec0b40ab4ef3bae09e54c8c078e387012bc1bae20e926c69d79aadaf56dec3943af362d4d323e5fd8d813d6f437875b722bc259f014aa4bb254d843779d3df6666e681438bff6a6420c59d831fbe672dce428e0cd4ad33a2a918c594e5f3d18e85cc0488fcb82bb5c66b1c0d13b6b0b9c043fdadbf6f90ea171a763c38b70027afdc5289b3b709b46ebe0873efbeb9441ad48a722b5f58f35207a0050e94c03eff1This result is
PASS_WITH_SEMANTIC_EXCEPTIONSunder the skill terminology. Raw baseline/candidate SHA values differ only because the baseline still serializes the KVBM G1–G4/offload/bandwidth configuration keys that this PR explicitly removes. Removing exactly those baseline-only retired keys produces the normalized semantic SHA shown above; no scheduler, routing, timing, request, lifecycle, evidence, or report-value difference remains.Every pressure record was fully readmitted. Aggregated rows used all workers 0–3 and recorded 5,000 immediate, zero queued placements. Disaggregated rows used both prefill and decode workers 0–1, recorded 5,000 complete handoffs, and recorded 10,000 immediate, zero queued placements across the two pools. The pinned main baseline requalified vLLM aggregated reuse at 4,917 requests; baseline and candidate agree exactly.
A supplemental 1,000-request preflight also added Round-robin to the four KV Router rows. All eight supplemental rows were internally deterministic and matched across revisions after the same retired-schema normalization. The updated 5,500-block vLLM disaggregated seed exposed a missing deferred-ready fairness path during extraction; commit
23518163ebrestores it, and both the 5,000-row authoritative row and the supplemental Round-robin/KV Router rows now match.This section reports the skill's authoritative correctness stage. The separate Linux 60-pair performance campaign was not run as part of this PR validation.
Dependency-firewall tests confirm that
aisimulate-corecontains no Dynamo or KVBM dependency. Existing Mooncake/KV Router benchmark callers retain their replay/artifact entrypoints.Related issues
main.Summary by CodeRabbit