Skip to content

refactor(aisimulate): extract generalized mocker engine and replayer - #12525

Merged
tedzhouhk merged 38 commits into
mainfrom
yongmingd/aisimulate-engine-replayer
Aug 17, 2026
Merged

tedzhouhk merged 38 commits into
mainfrom
yongmingd/aisimulate-engine-replayer

Conversation

@dreamtalen

@dreamtalen dreamtalen commented Aug 1, 2026 •

Copy link
Copy Markdown
Contributor

Summary

This PR extracts Dynamo-neutral simulation execution into AISimulate while keeping Dynamo-specific composition in Dynamo.

  • aisimulate-core is the public Rust crate. Its engine module owns vLLM, SGLang, and TensorRT-LLM scheduler simulation, framework-local G1 KV accounting, preemption, timing, single-rank execution, and attention-DP grouping.
  • aisimulate-core::replay owns deterministic virtual time, logical-worker lifecycle, aggregated and disaggregated replay, placement/scaling seams, artifacts, canonical reports, and execution evidence.
  • The crate root exposes the stable EngineConfig, TimingModel, TimingModelConfig, ReplaySpec, ReplayReport, and Replayer facade. Advanced contracts remain under explicit engine and replay paths.
  • Dynamo owns Router and Planner composition and the Live Mocker driver, including transport, cancellation, publication, and production metrics.
  • Live Mocker and offline replay now drive the same attention-DP-aware Generalized Mocker Engine instead of maintaining separate scheduler implementations.
  • Python runners and the native aisimulate._runtime binding expose engine-only and Dynamo-composed replay to the CLI and Sweeper boundaries.
  • The retired KVBM-specific Mocker replay/offload paths are removed. No KVBM compatibility adapter is added; standalone production KVBM crates and runtime functionality are unchanged.

The dependency and module direction is:

Dynamo Router / Planner / Live adapters
                    ↓
        aisimulate-core::replay
                    ↓
        aisimulate-core::engine

aisimulate-core has no Dynamo, KVBM, or Tokio dependency.

Review surface

The raw GitHub diff substantially overstates the implementation delta because file extraction is counted on both the deleted and added sides. The following line-level analysis separates pre-existing code, mechanical edits, tests, and the extracted or reworked review surface.

image

Breaking changes

  • The Mocker-only KVBM G1–G4 replay/offload implementation is removed without a compatibility adapter. Production KVBM remains unchanged, but simulated G2–G4 coverage is no longer available in the current replay stack.
  • The corresponding python -m dynamo.mocker and replay options are removed: --g1-backend, --num-g2-blocks, --num-g3-blocks, --enable-g4-storage, --offload-batch-size, and the --bandwidth-g*-to-g*-gbps options. Framework-native G2 simulation is separate future work.
  • The dynamo-kv-replay-parity protocol now covers native G1. Historical KVBM parity campaigns must use a historical revision and its matching protocol.
  • Disaggregated attention-DP is rejected before materialization until rank-aware handoff identity is implemented in [Mocker] Support disaggregated attention-DP replay identity and handoff #12965.

Ownership and entrypoints

Layer Ownership
Stable Rust facade aisimulate-core
Scheduler, cache, G1 KV accounting, timing, attention-DP engine aisimulate-core::engine
Virtual-time replay, worker lifecycle, Agg/Disagg runtimes, reports/artifacts aisimulate-core::replay
Engine runner, traffic materialization, reporting, native PyO3 execution seam AISimulate Python package
Router/Planner policies and replay composition Dynamo
Live driver, transport, cancellation, publication, production metrics Dynamo Mocker

Current entrypoints are:

  • Engine-only replay: python -m aisimulate.replay
  • Dynamo-integrated replay: python -m dynamo.replay
  • Configuration sweeps: Sweeper(runner_factory=...).run(config) and the examples under aisimulate/examples/sweeper/

The two replay modules share their engine, topology, traffic, replay-mode, SLA, and output argument schema. The Dynamo entrypoint adds optional Router, Planner, AIC, and online integration; its no-adapter form remains compatible with engine-only replay.

Correctness hardening

The review and latest-main reconciliation preserve one execution path while adding focused safeguards:

  • Rebasing onto 29930c4cea incorporates fix(mocker): truncate disagg prefill output plan #13052; exact output-token plans remain authoritative, and disaggregated prefill truncates its local plan without mutating decode.
  • Grouped Live Mocker shutdown drains and acknowledges the final pass boundary before cancellation.
  • Production LiveEngine::submit() bursts are forwarded without limiting admission to one request per pass; the public compatibility request lane remains unbounded while grouped-engine command lanes remain bounded.
  • Attention-DP ranks report FPM duration at the modeled shared group boundary.
  • Aggregated replay dispatches placement releases from every topology callback.
  • SGLang supports disabled prefix caching while rejecting unsupported KV-token-ID and non-chunked-prefill controls.
  • vLLM cached-prefix admission state survives preemption and propagates consistently through replay artifacts and Dynamo completion usage.
  • SGLang cache metrics remain latched across grouped completion.
  • Disaggregated attention-DP and custom replay worker selectors fail early rather than executing unsupported semantics.

API walkthrough

Expand API walkthrough

Stable facade

aisimulate-core exposes the common stable surface at the crate root:

aisimulate_core::{
    EngineConfig,
    TimingModel,
    TimingModelConfig,
    ReplaySpec,
    ReplayReport,
    Replayer,
}

Advanced engine and adapter contracts use explicit aisimulate_core::engine and aisimulate_core::replay paths.

Overall flow

python -m aisimulate.replay
    ↓
EngineReplayRunner
    ↓
aisimulate._runtime
    ↓
Replayer<RoundRobinComposition>

python -m dynamo.replay / Sweeper
    ↓
DynamoReplayRunner
    ↓
Dynamo Router / Planner composition
    ↓
Replayer<DynamoComposition>
    ↓
GeneralizedMockerEngine<SchedulerRank>

Live Mocker does not use the virtual-time Replayer, but shares its grouped engine:

Offline Replayer ── virtual clock ──┐
                                    ├─ Generalized Mocker Engine
Live Mocker ─────── Tokio clock ────┘

Generalized Mocker Engine

Main files:

  • aisimulate/crates/core/src/engine/generalized/contracts.rs
  • aisimulate/crates/core/src/engine/generalized/engine.rs
  • aisimulate/crates/core/src/engine/scheduler/

SchedulerRank owns one rank's scheduler, KV accounting, preemption, and timing state. GeneralizedMockerEngine<C> owns attention-DP membership, readiness, pass barriers, and modeled group completion. Offline replay supplies virtual time; Live Mocker supplies Tokio/wall-clock events through a Dynamo-owned driver.

Replayer

Main files:

  • aisimulate/crates/core/src/replay/replayer.rs
  • aisimulate/crates/core/src/replay/spec.rs
  • aisimulate/crates/core/src/replay/engine.rs

Replayer<C: ReplayComposition> owns virtual time, the event queue, logical-worker lifecycle, Agg/Disagg runtime state, trace collection, artifacts, and reports.

  • Replayer::new(...) provides round-robin placement with no scaling.
  • Replayer::with_composition(...) accepts Dynamo-owned placement and scaling policies.
  • Replayer::run_with_artifacts(...) uses the shared aggregated runtime with an optional Replay-owned artifact sink.

Placement, scaling, engine, and invariant failures retain typed provenance.

Python runner and CLI boundary

Main files:

  • aisimulate/src/aisimulate/runner.py
  • aisimulate/src/aisimulate/replay/
  • components/src/dynamo/replay/simulation.py
  • components/src/dynamo/replay/main.py

EngineReplayRunner materializes a serializable ReplaySpec, calls aisimulate._runtime, and normalizes ReplayReport. DynamoReplayRunner implements the same Runner contract while constructing Dynamo Router and Planner providers.

python -m dynamo.replay composes the shared AISimulate base parser/lowering with Dynamo-only extensions; it does not duplicate the engine-only schema.

Router, Planner, and Live adapters

Dynamo reuses the existing PlacementPolicy<Request> boundary, including round-robin and KvRouterPlacement, through ReplayComposition. Planner is injected through the scaling-policy seam.

Live Mocker constructs the same grouped engine as offline replay. Dynamo remains responsible for Tokio driving, transport, cancellation, output publication, and production metrics.

Validation

Final HEAD: 23518163eb

Rust

  • cargo test -p aisimulate-core -p dynamo-mocker -p dynamo-mocker-backend --no-fail-fast
    • aisimulate-core: 449 tests passed
    • dynamo-mocker: 194 tests passed
    • dynamo-mocker-backend: 20 tests passed
  • cargo clippy -p aisimulate-core -p dynamo-mocker -p dynamo-mocker-backend --all-targets -- -D warnings
  • cargo fmt --all --check
  • pre-commit run --all-files
  • git diff --check
  • Root, Python-binding, and KVBM-binding cargo metadata --locked checks passed.
  • A tracked-only EPP/ext-proc Docker workspace smoke passed cargo metadata --locked; both builders now copy aisimulate/crates/ alongside lib/ before Cargo resolves the root workspace.
  • A tracked-only Planner wheel context containing every root Cargo workspace member passed cargo metadata --locked and built the native AISimulate wheel; a regression test now enforces complete workspace COPY coverage.

Focused regressions cover grouped shutdown, production-lane submit batching, attention-DP FPM normalization and rank identity, vLLM/SGLang MTP lifecycle boundaries, cached-prefix propagation, SGLang cache-metric latching, bounded same-timestamp retry, pass-end KV visibility, topology release evidence, artifact capture, unsupported topology/selector rejection, and SGLang prefix-caching behavior.

Python and packaging

  • The dedicated Planner test image builds for both linux/amd64 and linux/arm64, is readable as the CI runner identity (1001:1001), and collects the intended 25 optional AISimulate/Planner tests.
  • Native Linux arm64 validation with current AISimulate and ai_dynamo_runtime wheels passed the exact Planner selector (25 passed, 134 deselected), the online KV Router CLI regressions (2 passed), and SGLang AIC parity (6 passed, 6 deselected).
  • The Planner image's two standalone wheel/build-context source-contract tests passed.
  • 173 targeted Runner, replay CLI, traffic, Sweeper, and Dynamo replay tests passed.
  • The standalone AISimulate wheel builds and imports aisimulate._runtime on Python 3.12.
  • Wheel metadata confirms no ai-dynamo dependency and no console script.
  • CODEOWNERS coverage remains 100%.

Replay parity

The authoritative native-G1 correctness campaign compared candidate 23518163eb with pinned merge-base main 975d6a15a6 using the first 5,000 requests from the canonical Mooncake trace. The frozen slice SHA-256 is 3892ae19ae480b643155f0c6b9d798591cbe2e73bec6a0fa5ae3d3bc0332fb8a, matching the dynamo-kv-replay-parity protocol.

The four skill-aligned KV-aware rows used the current qualified internal-polynomial seeds. Each baseline and candidate row ran twice in a separate process; every revision/row pair produced one deterministic digest, completed 5,000/5,000 requests, and matched after accounting for the intentionally retired KVBM configuration schema:

Engine Topology Baseline raw SHA-256 Candidate raw / normalized semantic SHA-256 Pressure Requests with reuse
vLLM Aggregated cbda006db3b1b4c7fe4eda15412469aa237b5ada44d1aaa39c5261b136ef6180 b69f998ec1978f5bf00f172d7648467af730e29cb57d7e42210664fbc220ad27 1 preemption 4,917
vLLM Disaggregated dd693851bdc43cff67fe049ef77cf0f4d41e85c869a0bfd995a85d72147e6ec0 b40ab4ef3bae09e54c8c078e387012bc1bae20e926c69d79aadaf56dec3943af 14 preemptions 4,988
SGLang Aggregated 362d4d323e5fd8d813d6f437875b722bc259f014aa4bb254d843779d3df6666e 681438bff6a6420c59d831fbe672dce428e0cd4ad33a2a918c594e5f3d18e85c 1 retraction 4,840
SGLang Disaggregated c0488fcb82bb5c66b1c0d13b6b0b9c043fdadbf6f90ea171a763c38b70027afd c5289b3b709b46ebe0873efbeb9441ad48a722b5f58f35207a0050e94c03eff1 14 retractions 4,990

This result is PASS_WITH_SEMANTIC_EXCEPTIONS under the skill terminology. Raw baseline/candidate SHA values differ only because the baseline still serializes the KVBM G1–G4/offload/bandwidth configuration keys that this PR explicitly removes. Removing exactly those baseline-only retired keys produces the normalized semantic SHA shown above; no scheduler, routing, timing, request, lifecycle, evidence, or report-value difference remains.

Every pressure record was fully readmitted. Aggregated rows used all workers 0–3 and recorded 5,000 immediate, zero queued placements. Disaggregated rows used both prefill and decode workers 0–1, recorded 5,000 complete handoffs, and recorded 10,000 immediate, zero queued placements across the two pools. The pinned main baseline requalified vLLM aggregated reuse at 4,917 requests; baseline and candidate agree exactly.

A supplemental 1,000-request preflight also added Round-robin to the four KV Router rows. All eight supplemental rows were internally deterministic and matched across revisions after the same retired-schema normalization. The updated 5,500-block vLLM disaggregated seed exposed a missing deferred-ready fairness path during extraction; commit 23518163eb restores it, and both the 5,000-row authoritative row and the supplemental Round-robin/KV Router rows now match.

This section reports the skill's authoritative correctness stage. The separate Linux 60-pair performance campaign was not run as part of this PR validation.

Dependency-firewall tests confirm that aisimulate-core contains no Dynamo or KVBM dependency. Existing Mooncake/KV Router benchmark callers retain their replay/artifact entrypoints.

Related issues


Open in Devin Review

Summary by CodeRabbit

  • New Features
    • Added an engine-neutral replay system supporting aggregated and disaggregated simulations across vLLM, SGLang, and TensorRT-LLM.
    • Added Python and Sweeper runner integration, configurable timing models, workload and trace loading, scaling policies, handoff simulation, KV-cache events, and optional per-request or raw reports.
    • Added attention data-parallel routing and grouped live-engine support.
  • Bug Fixes
    • Added validation for invalid configurations, malformed traces, unsupported topologies, and runtime errors.
  • Documentation
    • Updated replay, engine, compatibility, and configuration guidance.

@copy-pr-bot

copy-pr-bot Bot commented Aug 1, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added refactor documentation Improvements or additions to documentation planner container actions labels Aug 1, 2026
@dreamtalen
dreamtalen force-pushed the yongmingd/aisimulate-engine-replayer branch 15 times, most recently from 9164057 to dfcce49 Compare August 7, 2026 19:54

@tedzhouhk tedzhouhk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I finished reviewing the public API. The engine/replayer extraction and the Engine/Dynamo composition boundaries look good to me; I do not have API concerns with those parts.

I am requesting changes specifically for the user-facing CLI. The proposed aisimulate predict interface mixes several abstraction levels and is difficult to reason about. A few examples:

  • --isl and --osl are top-level CLI flags, while concurrency, request_count, request_rate, and arrival controls for the same traffic model live in YAML, sometimes at different YAML levels.
  • Trace input comes through a separate --traffic YAML, while config.workload can override it. This gives one workload multiple input sources with non-obvious precedence.
  • Deployment is split between CLI flags (model, backend, system, tp-size, and replicas) and config fields (num_workers, engine_args, and prefill/decode), with multiple aliases for some fields.
  • concurrency appears both inside workload and as a top-level ReplaySpec field. The latter is understandable as a Sweeper-resolved value for KV-load candidates, but exposing that intermediate contract through a concrete one-shot CLI is confusing.
  • The existing replay CLI's domain split—traffic, Router, prefill/decode engines, and Planner—is easier for users to understand.

Please defer the CLI changes from this PR for now. I suggest landing the engine/replayer/API extraction without establishing this new public CLI contract. Keep single-run replay and Sweeper as two separate user-facing paths for the moment; do not partially combine them through aisimulate predict until we have discussed and agreed on a coherent, typed CLI/config model. We can introduce the consolidated predict/sweep surface in a follow-up after that design discussion.

@tedzhouhk tedzhouhk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent adversarial verification confirmed the two blocking P1s and the three scoped P2s below. I am not including the previously considered mid-pass CancelDestination issue because both shipped drivers defer that command until the pass has completed.

Comment thread lib/mocker/src/generalized_live.rs Resolved
Comment thread aisimulate/src/aisimulate/runner.py Resolved
Comment thread aisimulate/crates/replay/src/agg.rs Resolved
Comment thread aisimulate/crates/engine/src/config.rs Resolved
@dreamtalen
dreamtalen force-pushed the yongmingd/aisimulate-engine-replayer branch from dfcce49 to 4031f29 Compare August 7, 2026 22:52
@dreamtalen

Copy link
Copy Markdown
Contributor Author

@tedzhouhk Thanks, removed the new aisimulate predict user-facing CLI and single-run replay remains python -m dynamo.replay in this PR.

Base automatically changed from hzhou-codex/spica-aisimulate-decoupling to main August 7, 2026 23:08
@dreamtalen
dreamtalen force-pushed the yongmingd/aisimulate-engine-replayer branch from 4031f29 to 40c1791 Compare August 7, 2026 23:58
@dreamtalen
dreamtalen marked this pull request as ready for review August 7, 2026 23:59
@dreamtalen
dreamtalen requested a review from a team as a code owner August 7, 2026 23:59
@saturley-hall

Copy link
Copy Markdown
Member

/ok to test 6a6556e

Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
@saturley-hall

Copy link
Copy Markdown
Member

/ok to test 293c9c1

Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
@saturley-hall

Copy link
Copy Markdown
Member

/ok to test b1172bf

Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
@saturley-hall

Copy link
Copy Markdown
Member

/ok to test 6b3d991

@saturley-hall saturley-hall left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the passing PR workflow I think this satisfies the minimal ops requirements.

Note we will need to file a follow-up PR to enable aisimulate to go out with the nightly dynamo build and update the dependency to that nightly version so that the latest work is verifiable.

Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
@saturley-hall

Copy link
Copy Markdown
Member

/ok to test f1e68

Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>
@saturley-hall

Copy link
Copy Markdown
Member

/ok to test 561ca5e

Signed-off-by: Harrison King Saturley-Hall <hsaturleyhal@nvidia.com>

Copy link
Copy Markdown
Contributor

/ok to test daeb2ac

@tedzhouhk
tedzhouhk merged commit a5228ae into main Aug 17, 2026
126 of 128 checks passed
@tedzhouhk
tedzhouhk deleted the yongmingd/aisimulate-engine-replayer branch August 17, 2026 20:55
@leigao97

Copy link
Copy Markdown

Hi @dreamtalen,

I've been using the kvbm offloading feature in DynoSim. Since this feature was removed in this refactor, would it be added back later?

Thanks.

@dreamtalen

Copy link
Copy Markdown
Contributor Author

Hi @leigao97, we plan to restore kv offload simulation in the AISimulate repo https://github.com/ai-dynamo/aisimulate, but not through KVBM, since KVBM is being sunset from Dynamo.

Could you share which specific KVBM offloading features or workflows you have been using? That would help us prioritize the replacement. Thanks!

@leigao97

Copy link
Copy Markdown

Thank you, @dreamtalen, for the quick reply. I was primarily using the KVBM offloading feature to simulate three-tiered (HBM/DRAM/SSD) prefix cache offloading with configurable capacity and bandwidth, similar to the HiCache design in SGLang. Having an accurate simulation of these multi-tiered offloading behaviors in AISimulate would be incredibly valuable for my workflows.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

actions backend::trtllm Relates to the trtllm backend container documentation Improvements or additions to documentation planner refactor size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants