Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
af724ee
feat(capture): restore immutable attempt v1 client protocol
Wilboludriver Nov 9, 2026
a4804d6
feat(capture): migrate bounded HS cache and assembly controller
Wilboludriver Nov 9, 2026
a77aec7
feat(capture): distinguish atomic capacity refusal from prompt failure
Wilboludriver Nov 9, 2026
52c78a5
feat(runtime): integrate native HS attempt ownership and continuous r…
Wilboludriver Nov 9, 2026
8fddf46
feat(capture): integrate fenced HS ownership with native outputs
Wilboludriver Nov 9, 2026
37acad3
fix(capture): validate full HS metadata and preserve capacity retry i…
Wilboludriver Nov 9, 2026
097c954
fix(runtime): fence new receives before transport shutdown
Wilboludriver Nov 9, 2026
9fab394
fix(capture): bound producer leases and coalescing by attempt capacity
Wilboludriver Nov 9, 2026
68ec940
fix(capture): vote local TP failures before nested collectives
Wilboludriver Nov 9, 2026
7cfa9b6
fix(capture): retain resolved TP rank in HS publication diagnostics
Wilboludriver Nov 9, 2026
0aa4821
fix(capture): gate attention DP on resolved topology at startup
Wilboludriver Nov 9, 2026
5b37de5
perf(hs): skip disabled-cache adoption and empty-lease hashes
Wilboludriver Nov 10, 2026
150cb43
feat(hs): add bounded opt-in publication boundary diagnostics
Wilboludriver Nov 10, 2026
544d097
diag(hs): split bounded materialization timings behind opt-in flag
Wilboludriver Nov 10, 2026
2751f90
fix(hs): avoid duplicate token scans on validated hash path
Wilboludriver Nov 10, 2026
1fbaac6
fix(diag): bound controller timing events by sample policy
Wilboludriver Nov 10, 2026
391c34d
Retain publication sources when executor acceptance is unknown
Wilboludriver Nov 10, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions plugins/sglang-spec-capture/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,3 +180,82 @@ with RDMA NICs. Without it, Mooncake installs its RDMA transport instead.
| `SGLANG_SPEC_CAPTURE_ARENA_BYTES` | Arena size in bytes. Required with `MOONCAKE_PROTOCOL=nvlink`. |
| `SGLANG_SPEC_CAPTURE_CONTROL_PORT` | Port of the NVLink free endpoint (default: ephemeral). |
| `MOONCAKE_*` | Mooncake client settings. |

## Opt-in HS capture protocol v1

This migration additionally needs SGLang's native
`managers.prefix_admission.CAPTURE_ADMISSION_PROTOCOL == 1` companion extension.
It does not apply the removed legacy capture patch. Enable
`SPECFORGE_HS_CAPTURE=1` to negotiate server-issued attempt reservations at
`POST /spec_capture/control`; set `SPECFORGE_HS_CACHE=1` to reuse hidden-state
prefixes. `HS_CAPTURE=1, HS_CACHE=0` keeps the same complete-output, byte-budget
and immutable-attempt protocol while forcing a zero prefix. The ordinary
non-negotiated capture path remains available when both are off.

| Variable | Default |
|---|---:|
| `SPECFORGE_HS_CHUNK_TOKENS` | 256 |
| `SPECFORGE_HS_CACHE_BYTES` | 1073741824 |
| `SPECFORGE_HS_PINNED_BYTES` | 268435456 |
| `SPECFORGE_HS_OUTPUT_BYTES` | 536870912 |
| `SPECFORGE_HS_DEVICE_BYTES` | 268435456 |
| `SPECFORGE_HS_MAX_WORKERS` | 4 |
| `SPECFORGE_HS_ATTEMPT_CAPACITY` | 4096 |

All limits must be positive. Pinned staging uses one retained arena, conservatively
rounded down to a power of two, with aligned independent reservation slots.
Cache adoption charges complete underlying storage, including unmatchable tails.
Output, pinned and device limits are separate. Hard oversize fails explicitly;
temporary admission pressure permits only the native transaction's single
fully-rolled-back reclaim/retry. Atomic registry-capacity rejection returns
`error_code=capture_capacity_declined, reservation_granted=false` before granting
any attempt in that begin operation.

HS capture supports complete causal text, nonchunked prefill, PP1/DP1, no
offload/session/LoRA/weight updates, and host publication with Mooncake hard pin.
Set `SGLANG_SPEC_CAPTURE_GPU_PUT=0` when using RDMA host staging. Reuse additionally
requires deterministic batch-invariant inference, enabled radix, and an exact
selected recurrent checkpoint backend. Unsupported modes fail startup rather
than changing capture semantics. Capability metadata advertises these limits.

The content namespace binds model/config/load epoch, ordered raw auxiliary
layers, post-norm last hidden state, dtype/layout, causal position/attention,
TP/KV/checkpoint/backend and deterministic semantics. Tokens form complete parent
hash chains; sample/attempt IDs and loss masks do not enter content keys.
Passthrough data remains part of the registered request digest.

Each external attempt has immutable keys
`store/sample/g1/attempt/server_epoch/reservation_token/feature`. The native
observer retains independently owned suffix snapshots; its auxiliary output
carries controller jobs and performs no second D2H. Assembly reconstructs every
row `[0,N)` from the leased prefix and the current `[L,N)` suffix. Scheduler safe
points batch ready samples into the native sink, then resolve native deferred
responses. A PUBLISHED certificate is issued only after exact key/schema/digest
validation and source-reader drain. Sink completion permits cache adoption; it
does not retire storage retained by cache leases. Unknown DMA/native transfer
drain quarantines sources, registrations and charges until an external drain
proof exists. Timeouts do not authorize reuse.

`tests.test_runtime.test_hs_runtime` covers the migrated CPU core;
`tests.test_runtime.test_hs_plugin` covers native-output wiring with a CPU buffer
store; `test_hs_controller_parallel_cuda` requires actual CUDA/pinned copies.
CPU fixtures do not certify numerical model, real Mooncake, scheduler TP or
training equivalence.

Internal TP control/admission collectives use named local phases: owner/poll,
writer payload construction, received-state application, target lookup and
permit validation all vote local errors before entering a later collective.
Ranks agree request membership and control branches before dispatch. Startup
epoch/model/schema/core failures use the same protocol. `poll` and `lifecycle`
themselves remain local because native completion paths may call them only on
the writer. Begin operations preflight a separate registry on every rank;
only unanimous capacity refusal of uncommitted trials can emit the typed
`reservation_granted=false` proof. A rank disagreement or partial commit fails
closed without that proof.

`tests.test_runtime.test_hs_tp_protocol` drives the real runtime and companion
SG `PrefixAdmissionCoordinator.phase` with two CPU scheduler threads and a
blocking operation-order rendezvous. Set `SPECFORGE_HS_TEST_SG_SOURCE` to the SG
checkout when the installed editable still points at an upstream baseline.
This detects mismatched collective sequences in the tested faults; actual
distributed TP, CUDA and transport failure validation remains separate.
12 changes: 12 additions & 0 deletions plugins/sglang-spec-capture/specforge_sglang_capture/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,14 @@ def register() -> None:
HookRegistry.register(
f"{_SCHEDULER}.get_internal_state", _capture_internal_state, HookType.AFTER
)
from specforge_sglang_capture.hs.config import HSConfig
if HSConfig.from_env().capture:
from specforge_sglang_capture.hs.http import install_control_route
HookRegistry.register(
"sglang.srt.entrypoints.http_server.set_global_state",
install_control_route,
HookType.AFTER,
)


def _capture_streamer_class(streamer_class, scheduler, *args, **kwargs):
Expand All @@ -69,6 +77,10 @@ def _capture_internal_state(result, scheduler, *args, **kwargs):
result.internal_state["spec_capture_backend"] = "plugin"
result.internal_state["spec_capture_method"] = runtime.capture_method
result.internal_state["spec_capture_aux_layer_ids"] = runtime.sink.aux_layer_ids
hs = getattr(runtime, "hs", None)
if hs is not None:
result.internal_state["spec_capture_hs_protocol"] = 1
result.internal_state["spec_capture_hs_cache"] = hs.enabled
if runtime.is_writer:
control = runtime.sink.control_endpoint()
if control is not None:
Expand Down
60 changes: 53 additions & 7 deletions plugins/sglang-spec-capture/specforge_sglang_capture/capture.py
Original file line number Diff line number Diff line change
Expand Up @@ -117,12 +117,19 @@ def collect_capture_items(
if spec is not None:
error = spec.get("_parse_error")
prompt_len = len(req.origin_input_ids)
if error is None and prefix_len:
plan = getattr(req, "_specforge_hs_plan", None)
if error is None and plan is not None:
if (prefix_len != plan.L or length != plan.N - plan.L
or prompt_len != plan.N or plan.execution.request_id != req.rid):
error = "the native capture span differs from its committed HS plan"
elif error is None and "protocol_version" in spec:
error = "HS capture has no committed native admission plan"
elif error is None and prefix_len:
error = (
f"{prefix_len} prompt tokens were served from the prefix "
"cache; spec capture needs a full prefill"
)
elif error is None and length != prompt_len:
elif error is None and plan is None and length != prompt_len:
error = (
f"the prefill covered {length} of {prompt_len} prompt tokens; "
"launch the server with --chunked-prefill-size -1"
Expand Down Expand Up @@ -185,9 +192,25 @@ def after_forward(self, batch, forward_batch, logits_output, *, can_run_graph):
),
)
checked.append(item)
return CaptureDeviceOutput.from_rows(
self.runtime, checked, aux, last_hidden, copy=can_run_graph
)
jobs, native_items = [], []
reqs = {req.rid: req for req in batch.reqs}
for item in checked:
if self.runtime.hs is None or "protocol_version" not in item.spec:
native_items.append(item)
continue
req = reqs[item.rid]
if item.error is None:
try:
jobs.append(self.runtime.hs.snapshot(req, item, aux, last_hidden))
continue
except Exception as exc:
item = dataclasses.replace(item, error=str(exc))
self.runtime.hs.capture_error(req, item.error)
native_items.append(item)
output = CaptureDeviceOutput.from_rows(
self.runtime, native_items, aux, last_hidden, copy=can_run_graph)
output.hs_jobs = jobs
return output


def _select_rows(
Expand Down Expand Up @@ -235,44 +258,54 @@ def __init__(self, runtime, items, aux, last_hidden):
self.items = items
self.aux = aux
self.last_hidden = last_hidden
self.hs_jobs = []

@classmethod
def from_rows(cls, runtime, items, aux, last_hidden, *, copy):
items, (aux, last_hidden) = _select_rows(items, (aux, last_hidden), copy=copy)
return cls(runtime, items, aux, last_hidden)

def copy_to_host(self, copy_tensor):
if not self.items:
# HS controller owns the only D2H path and its bounded pinned arena.
return CaptureHostOutput(self.runtime, [], None, None, hs_jobs=self.hs_jobs)
if self.runtime.gpu_put:
# Device publication: the sink waits for everything enqueued so far.
ready = torch.cuda.Event()
ready.record()
return CaptureHostOutput(
self.runtime, self.items, self.aux, self.last_hidden, ready_event=ready
self.runtime, self.items, self.aux, self.last_hidden, ready_event=ready,
hs_jobs=self.hs_jobs,
)
return CaptureHostOutput(
self.runtime,
self.items,
copy_tensor(self.aux) if self.aux is not None else None,
copy_tensor(self.last_hidden) if self.last_hidden is not None else None,
hs_jobs=self.hs_jobs,
)


class CaptureHostOutput:
"""HostAuxiliaryOutput: publish finished capture requests, then hold them."""

def __init__(self, runtime, items, aux, last_hidden, *, ready_event=None):
def __init__(self, runtime, items, aux, last_hidden, *, ready_event=None, hs_jobs=()):
self.runtime = runtime
self.items = items
self.aux = aux
self.last_hidden = last_hidden
self.ready_event = ready_event
self.hs_jobs = hs_jobs

def _rows(self, tensor, item):
if tensor is None:
return None
return tensor[item.start : item.start + item.length]

def consume(self, batch, commits) -> None:
if self.hs_jobs:
self.runtime.hs.consume(self.hs_jobs, batch)
self.hs_jobs = []
reqs = {req.rid: req for req in batch.reqs}
samples: List[Sample] = []
held = []
Expand Down Expand Up @@ -433,6 +466,7 @@ def __init__(
self.results: Dict[str, Dict[str, Any]] = {}
self.pending = PendingCaptures(self, max_pending_batches)
self.observer = CaptureObserver(self)
self.hs = None


def install(scheduler) -> CaptureRuntime:
Expand Down Expand Up @@ -485,6 +519,18 @@ def install(scheduler) -> CaptureRuntime:
)
method = features.aux_hidden_state_capture
runtime.capture_method = str(getattr(method, "name", method)).lower()
from specforge_sglang_capture.hs.config import HSConfig
hs_config = HSConfig.from_env()
if hs_config.capture:
from sglang.srt.managers.prefix_admission import CAPTURE_ADMISSION_PROTOCOL
from specforge_sglang_capture.hs.runtime import HSCaptureRuntime

if CAPTURE_ADMISSION_PROTOCOL != 1:
raise RuntimeError("HS capture requires native admission protocol v1")
if getattr(scheduler, "prefix_admission_observer", None) is not None:
raise RuntimeError("another native prefix admission observer is already installed")
runtime.hs = HSCaptureRuntime(scheduler, runtime, hs_config)
scheduler.prefix_admission_observer = runtime.hs
model_runner.forward_observer = runtime.observer
scheduler.output_streamer.spec_capture_results = runtime.results
scheduler._specforge_capture_runtime = runtime
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
"""Single-server content-keyed raw feature cache, frozen Plan A v1 API.

block_keys(namespace, tokens) returns complete canonical chunk identities.
stats() returns bounded storage/reservation and lifecycle counters.
"""

from .cache import PrefixFeatureCache
from .controller import CaptureController
from .protocol import (
AdmissionBudgetError,
CaptureCancelledError,
CaptureCompletion,
CaptureContractError,
CapturePlan,
CapturedSpan,
ContentBlockKey,
DrainHandle,
ExecutionIdentity,
FeatureSchema,
FeatureSpec,
FillReservation,
MaterializedSample,
OutputReservation,
PrefixLease,
)

__all__ = [
"AdmissionBudgetError",
"CaptureCancelledError",
"CaptureCompletion",
"CaptureContractError",
"CaptureController",
"CapturePlan",
"CapturedSpan",
"ContentBlockKey",
"DrainHandle",
"ExecutionIdentity",
"FeatureSchema",
"FeatureSpec",
"FillReservation",
"MaterializedSample",
"OutputReservation",
"PrefixFeatureCache",
"PrefixLease",
]
Loading