Skip to content

Add opt-in teacher hidden-state prefix caching - #1188

Draft
Wilboludriver wants to merge 17 commits into
sgl-project:mainfrom
Wilboludriver:feat/hs-prefix-cache
Draft

Wilboludriver wants to merge 17 commits into
sgl-project:mainfrom
Wilboludriver:feat/hs-prefix-cache

Conversation

@Wilboludriver

Copy link
Copy Markdown

Motivation

Reuse teacher hidden states for repeated prefixes during online training while preserving complete feature outputs.

Modifications

  • Add an opt-in, bounded per-Teacher HS cache with native prefix admission and suffix capture.
  • Preserve request ownership, byte budgets, cancellation and safe publication/cleanup, including uncertain writer completion.
  • Include supporting rollout/polling controls and optional prompt-length and optimizer compatibility settings.

P2/P3 experiments, C2 and the planned Trainer-side content pool are outside this PR.

Related Issues

Requires a SGLang companion for native prefix admission and FP32 GDN checkpoint snapshots; that companion is not published yet. Related upstream precision fix: sgl-project/sglang#48682.

Accuracy Test

Existing correctness and training validation used pinned revisions: SpecForge 391c34d4 / SGLang d3dd55e5, based on upstream 224b36bd / c757c4bb. This includes scoped CPU lifecycle/fault checks, TP2 numerical comparisons and two-node RDMA training.

This draft opens the implementation for early feedback. Adaptation to current upstream main and its CI remain pending.

Benchmark & Profiling

Historical measurements use the pinned stack above. Exact configurations and reproducible results will be documented before marking this PR ready for review; no current-main performance claim is made.

Checklist

  • Format your code according to the Code Formatting with Pre-Commit guide.
  • Add unit tests as outlined in the Running Unit Tests guide.
  • Update documentation / docstrings / example tutorials as needed.
  • Provide throughput / latency benchmark results and accuracy evaluation results as needed.
  • For reviewers: remove unrelated co-author attribution when merging.
  • Optional: discuss the PR in the project Slack channel.

Source: SpecForge 0b3e00a:specforge/inference/capture_protocol.py, copied verbatim. Preserves server-issued attempt fences and publication transfer certificates for native capture integration.
Extract pure cache/protocol/controller and focused runtime/CUDA ownership tests from frozen 0b3e00a hs-prefix-cache.patch. Imports now belong to the native capture plugin; no legacy engine capture bridge or backend is restored. Retains atomic coverage leases, actual-storage cache adoption, disjoint pinned arena slots, byte admission, complete feature materialization and quarantine.
Only the explicit begin/begin_batch capture_capacity_declined response with reservation_granted=false authorizes backpressure retry. Other errors remain fail-closed under the existing protocol.
…ollout

Select generic HS semantics from 0b3e00a, keeping plugin-only capture and upstream durable ACK/receive pools. Explicit hs_capture/hs_prefix_cache negotiate immutable attempt ownership; coalesced begin retains independent completion and bounded byte credits. Atomic capacity refusal retries the same lease without prompt failures. Fenced failed reads quarantine storage and registration until owned transport shutdown.

Retain generic full-record filtering and optimizer/scheduler/polling compatibility. Native managed capture and live gate keep official version checks enabled. Requires the separately migrated capture_protocol.py v1 plus CaptureCapacityDeclined from the plugin layer.

Validation: 221 CPU tests, 219 pass / 1 skip / 1 expected failure; exact dirty source manifest in migration tests/p3-cpu-r2. No GPU numerical or production acceptance inferred.
Implements frozen interface v1: plugin-owned scheduler callbacks, atomic attempt registry/control routing, exact model/schema namespaces, compact suffix snapshots, sole bounded controller D2H, complete N-row assembly, batched sink publication and native deferred completion. Sink uses immutable attempt keys, drains argument references before cache adoption, and retains sources/registrations/charges for unknown native write drain. Adds focused CPU adversarial contracts and explicit supported-mode metadata. Sources: frozen 0b3e00a attempt/control ownership semantics; SF224b36bd native observer/aux-output/completion/sink remains the integration path. AST and whitespace checks only; independent CPU/GPU tests pending.
…dentity

Verify request digest and exact feature names/shapes/dtypes before adopting negotiated output. Permit a proven-refused lease to coalesce with fresh registrations without reissuing its attempt identity. Managed launch explicitly sets HS switches and never inherits accidental opt-in. Use an interpreter forwarding fixture compatible with relocatable Python wrappers.

Validation: CPU suite223 (221 pass/1 skip/1 expected failure); native-HS/launch101 pass. Real GPU and production validation pending.
Closes SF-TP-01 by synchronizing named local phases before and after internal startup, transfer, request, lookup, resolve, read and cleanup collectives. Native/HS prepare membership is agreed before transfer synchronization. Begin operations use isolated registry trials, agreed permit identities and voted reply construction before any actual grant; only unanimous uncommitted capacity declines emit typed no-grant proof. Poll and lifecycle remain free of collectives. Strengthens shared-sink cancellation to assert the same publication Future for two requests and adds actual-runtime/SG-phase threaded CPU contracts over 35 fault/success scenarios. CPU HS/core/protocol subset: 59 passed, no skips, CUDA uninitialized; real distributed/GPU/transport validation remains pending.
Snapshot tp_rank alongside tp_size from SGLang runtime_context.get_parallel, the topology used by native capture writer election. Preserve the local rank in each committed plan so history, PUBLISHED results and transfer certificates retain the required diagnostic without changing physical identity or ownership. Tests check the resolved-context path, cache-off/result/certificate fields and separate rank0/rank1 committed diagnostics without a scheduler.tp_rank attribute. Relevant CPU subset: 9 passed, no skips, CUDA uninitialized; actual GPU validation remains with the validation owner.
Use ParallelContext.attn_dp_size instead of the deprecated enable_dp_attention record field, which remains false after SG resolution. Replica DP, PP and CP startup gates are unchanged. The new CPU regression invokes the actual _model_identity with the real get_parallel/ParallelContext override: attention-DP2 plus legacy false is rejected before checkpoint/backend validation; ordinary TP1/TP2 attention-DP1 and existing replica/PP/CP boundaries are checked. Six focused gate/rank/publication contracts passed with CUDA uninitialized; no GPU, environment or integration checkout changes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant