Repository navigation
Crash the server when error or OOM happens - #514
Merged
Merged
Conversation
timethink
pushed a commit
to timethink/sglang
that referenced
this pull request
Mar 9, 2025
vschandramourya
pushed a commit
to vschandramourya/sglang
that referenced
this pull request
Feb 3, 2026
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…gl-project#506 Closes the four A2 findings from AUDIT_506_unswept_axes.md's state-changing HTTP endpoint sweep: - A2-F1: /hibernate no longer accepts an unvalidated hibernate_dir override from the request body (path-confinement check, new utils/path_confinement.py) and drops the GET method -- POST only. - A2-F2/F3: the fork's own state-changing routes now carry an explicit auth requirement instead of relying on ADMIN_FORCE placement; CORS credentials no longer combine with a wildcard origin. - A2-F4: the registry control plane (registry/launch.py, registry/http_api.py) requires auth before accepting an argv-driven engine launch. - A2-F5: video_enhance/mux.py's ffprobe/ffmpeg source_url gets a scheme allowlist and no longer reflects raw subprocess stderr to the caller (subprocess_failure() wraps it into a MuxError that omits the detail). - /v1/files (training/service.py) now guards on the training-enabled flag before writing to disk. Clean auto-merge -- FEATURE_CATALOG.md and server_args.py both merged without conflict (neither had any other change on this line since the fceaec3 base). One follow-up fix in the same commit: ruff flagged 2 NEW UP037 findings in the branch's own new subprocess_failure() (quoted `"bytes | str | None"` / `"MuxError"` annotations that don't need quoting -- MuxError is defined earlier in the file, no forward-reference issue) -- autofixed with `ruff check --select=UP037 --fix`, confirmed clean after. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - test/registered/unit/entrypoints/ + registry/ + training/ + workbench/: 605 passed, 61 subtests. - test/registered/video_enhance/ + unit/server_args/: 1186 passed, 4 skipped, 266 subtests. - ruff check --select=F401,F821,UP037: clean after the mux.py autofix. No ruff format run on planner/* or other listed subtrees (CLAUDE.md sgl-project#514-lesson rule) -- not applicable here anyway, no planner files touched. - codespell clean on all 11 touched/added files.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…gl-project#506 axis 3) Incomplete-cache-key family (sgl-project#241/sgl-project#513): a persisted artifact whose CONTENT depends on a dimension its KEY does not carry is a silent wrong hit, not a miss. sgl-project#513 closes four instances the sgl-project#506 axis-3 sweep found: (a) rigmon/card_probe.py wrote a correct key (sorted UUIDs, driver, probe version) but both readers (planner/solver_api.py, planner/rig_profile_source.py) took the newest file by mtime instead -- card_probe.matching_cached_probe_json() is now the single reader, a non-match is a miss. (b) HiCache page keys carried tp_rank/tp_size, which under uneven TP does not determine a rank's kv-head count; --rank-tp-ratio/--rank-kv-ratio now enter compute_model_identity_hash (only when set, so even-TP rigs keep their persisted pages). (c) The measured-KV-budget fingerprint omitted attention_backend, disable_cuda_graph, dtype and enable_hierarchical_cache -- all four change the post-capture VRAM leftover it records; added on the file's own only-when-non-default convention. (d) planner/graphmem.py keyed the NUMBER of capture batch sizes, so two different --cuda-graph-bs lists of equal length shared an anchor; the key is now versioned (ANCHOR_KEY_VERSION v2) and carries the list, attention backend and page size. Two existing graphmem tests had been passing BECAUSE of the collision -- fixed alongside. One real conflict, as expected (two branches both add a new paragraph to the very start of FEATURE_CATALOG.md's §12, right after the header): kept BOTH additive paragraphs, sgl-project#510's "Unauthenticated-state-change family" first (already on the line), sgl-project#513's "Incomplete-cache-key family" appended directly after. uneven_perf.py's fingerprint-fields addition auto-merged clean (self-contained tail addition, no other change on this line touched that function). All 17 sections verified intact. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - test/registered/unit/planner/ (HTSGLANG_TEST_MODEL_DIR set): 2225 passed, 1 skipped, 352 subtests (up from 2223 pre-merge, +2 from test_graphmem.py's new methods). - test/registered/unit/mem_cache/ + unit/rigmon/ (excluding test_hicache_nixl_storage.py, which needs the external `nixl` package not installed in this venv -- environment gap, unrelated to sgl-project#513): 937 failed total, but EVERY failure traced to one untouched file, test_unified_radix_cache_unittest.py. Verified non-regression by direct comparison, not assumption: ran that file alone against both the pre-merge base (fceaec3) and the merged tree -- both produce byte-identical 851 failed / 16 passed / 648 skipped (diff of the two failing-test lists is empty). Root cause is environmental (RuntimeError: No accelerator (CUDA/XPU/HPU/NPU/MUSA/MPS) under this hermetic CPU-only invocation), not something sgl-project#513 touches or introduces. Every other file in the two directories passed clean. - test/registered/hicache/*.py (top-level, not unit/): needs a live server boot + model download (one hits a gated HF repo) -- out of scope for a hermetic merge check, unrelated to sgl-project#513, left unexercised. - ruff check --select=F401,F821,UP037: 5 findings, all confirmed pre-existing at fceaec3 and outside sgl-project#513's diff hunks (graphmem.py's unused `math` import, three card_probe.py `from_json` UP037 quotes, uneven_perf.py's long-standing Dict[str, List[float]] quote). No ruff format run on planner/* (CLAUDE.md sgl-project#514-lesson rule). - codespell clean on all 9 touched/added files.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…-wrongness sweep Eight commits, each closing or correcting a sgl-project#505 finding: - B-01: offload_movement splits the wave-in error path -- a destination release that already landed is not a failed retrieval; the item is only marked PARKED and its memtier booking only held when the bytes genuinely are still at the park target. - C-01: step 0 of the sgl-project#505-C-01 liveness-timeout task REFUTES its own ranking -- TTFT is outside the 90s budget (the streaming response is only built after generator.__anext__() returns, so the watchdog does not exist during queueing/prefill). The default is deliberately left unchanged; what sgl-project#514 pins instead is the ORDERING that makes that true (TimeToFirstTokenIsOutsideTheBudgetTest, proven can-fail by removing the pre-pull). Same lesson as sgl-project#500-B1: an audit finding is a hypothesis until its falsifier is red. - C-05: CONVENTION_bounding_defaults.md + a value-pinning test convention, with the first three defaults pinned against removal/halving. - A1-01: widens the draft-load completeness check and makes its non-reach audible instead of silent. - A2-04: PD topology returns UNKNOWN instead of fabricating a CUDA->NVML identity mapping when it cannot resolve one. - D3: scrubs the sgl-project#152 GitHub result-share route (was rendering the start command's argv verbatim) and binds the posted body to what the user actually previewed. - A2-03/A2-01: the kv-session-offload spill verdict is now rank-uniform (a group decision), and a carve-out that used to be assumed is now verified against the live state. - A2-05: the barlink transport fallback is a GROUP decision, not a per-rank one. Also PARTIALLY REFUTES the sgl-project#505-A2-05 damage story: the byte proof already reduces its verdict group-wide, so the three named sites desync into a hang, not a silent split; the genuine split window is narrower (a rank-local exception after a transport's last bring-up collective) and is what this fix actually closes. The broader desync class is untouched, recorded as a separate GPU-window ticket. Corrects AUDIT_505_silent_wrongness.md and FEATURE_CATALOG.md in place with the same append-only CORRECTION/REFUTED convention used throughout this line (blockquotes, nothing deleted). Clean auto-merge, no conflicts in either file -- all 17 catalog sections and every previously-preserved additive paragraph (two-axis §1, matrix doctrine §2, sgl-project#510/sgl-project#513/sgl-project#501 §12 families, MERGE-DUTY owner-sites + SITREP §12, NVENC §13) verified present. test_kvso_reclaim_decline_501.py (sgl-project#501's own test) picks up a 1-line change from sgl-project#514's kv_session_offload.py work -- sgl-project#514's base already contains sgl-project#501, so this is sgl-project#514 building on top of it, not a collision. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - Targeted run across disaggregation/, the new/changed test files, managers, distributed (barlink/gpu_battery), and liveness: 530 passed, 21 subtests, 1 failure (test_decode_queue_cleanup.py, untouched by sgl-project#514's diff; reproduced byte-identically on the pre-merge base 96c7dc5 -- same generic retry()-wrapped environmental failure seen throughout this line). - test/registered/unit/distributed/ (full): 24 failed, 2147 passed, 8 skipped, 721 subtests -- matches the established 24-failure baseline. - test/registered/unit/model_loader/ (full, since loader.py/weight_utils.py are touched): 32 failed, 229 passed, 9 skipped, 41 subtests on the merged tree vs 32 failed, 226 passed on the pre-merge base -- diffed the two failing-test-ID lists directly: BYTE-IDENTICAL. The three extra passes are sgl-project#514's own new/extended test coverage elsewhere in the directory. - ruff check --select=F401,F821,UP037: 2 findings (offload_movement.py:206 "DeviceOps", loader.py's unused _export_static_state import), both confirmed pre-existing at 96c7dc5 and outside sgl-project#514's diff hunks. No ruff format run (CLAUDE.md sgl-project#514-lesson rule -- ironically the same task number). - codespell: one hit, "unter" in test_gpu_battery_checks.py:1965 (a pre-existing German test-fixture string, `"reason": "unter der Marke"`, already present at 96c7dc5 and untouched by sgl-project#514's diff -- a pre-existing English-only-code violation, not introduced here).
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…pert stride; catch-all probe registration Three sgl-kernel csrc fixes, each with an off-GPU can-discriminate test (pure tree change -- wheel NOT built/installed here): - sgl-project#511 (audit sgl-project#506 axis 4, sgl-project#380 class): the GGUF MXFP4 numeric gate was loose enough that an all-zeros or sign-flipped output passed (atol=1.5, rtol=3e1 against RMS 5.1e3 outputs). Replaced with a tolerance DERIVED from the one physical error source (q8_1 activation rounding), plus spread/magnitude preconditions. Every gate in test_gguf_mxfp4_cuda.py now has an off-GPU test proving it rejects a zeroed and a sign-flipped output, including the refuted baseline executed as its own test. - sgl-project#512 (audit sgl-project#506 A1-1): the GGUF MoE MMQ kernel's expert-address byte stride (moe.cuh) was declared `int` (32-bit) while the call site already passed int64_t -- wraps negative once a rank's per-layer expert tensor passes 2 GiB (DSV4-Flash Q4_K, 256 experts, first bad local expert 227; TP=3 sharding is the only reason this rig never hit it). Widened to int64_t through all instantiations. - sgl-project#518: three GGUF capability probes (ggml_moe_get_block_size, ggml_mmvq_kq_tuned, ggml_mxfp4_native) were registered for a DEVICE dispatch key but take no tensor argument, so the dispatcher can never route them -- every call raised "no tensor arguments" before any kernel ran, on every arch. Serving never noticed because gguf.py had grown a python mirror around the raise. Fixed with the same keyless m.impl pattern apply_token_bitmask_inplace_cuda already uses. test_no_tensor_op_dispatch_518.py pins it twice: the dispatcher behaviour on throwaway in-process ops (executed, off-GPU, no wheel needed) and a ratchet over every TORCH_LIBRARY_FRAGMENT schema. FEATURE_CATALOG.md §12 conflict as expected (both this branch and the already-merged sgl-project#510/sgl-project#513 insert a new family paragraph immediately after the §12 header): kept all three additive paragraphs in sequence (Unauthenticated-state-change sgl-project#510, Incomplete-cache-key sgl-project#513, Unreachable-registration sgl-project#81/sgl-project#518, Byte-stride width sgl-project#109/sgl-project#112/sgl-project#512, Tolerance-that-cannot-fail sgl-project#380/sgl-project#511). All 17 sections and every other previously-preserved additive paragraph verified present. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree, no wheel rebuild): - The three sgl-project#511-touched test files directly: 29 passed, 14 skipped (the CUDA-gated MXFP4 kernel cases correctly skip without a rebuilt wheel; the off-GPU can-discriminate portions all run and pass). - test/registered/unit/quantization/ (full): 6 failed, 100 passed, 14 skipped, 27 subtests -- all 6 failures confined to test_gguf_dequant_scratch_budget.py, untouched by sgl-project#511's diff. - test/registered/unit/layers/ (full): 3 failed, 646 passed, 39 skipped, 995 subtests -- all 3 failures in test_deterministic_fp8_gemm.py / test_flashinfer_workspace_zeroing.py, both untouched by sgl-project#511's diff. - test/registered/unit/model_loader/ (full, since gguf.py is touched): 32 failed, 229 passed, 9 skipped, 41 subtests. Diffed the FULL failing-test-ID list (not just the tail) against the pre-sgl-project#511 tip (post-sgl-project#514): BYTE-IDENTICAL, 32 lines each side, empty diff. An earlier truncated `tail` comparison looked different due to test-order variance in what scrolled past -- re-verified against the complete list before trusting it. - ruff check --select=F401,F821,UP037: clean. - codespell: clean on all 9 touched/added files including the .cc/.cuh sources.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
…ough the declaration overlay Task sgl-project#465 fixture repair + sgl-project#499 hibernate fix, two commits: test(sgl-project#465): repairs four stale test fixtures across the line (7 red -> 0): test_decode_queue_cleanup.py, test_bar1_strand_is_opt_in.py, test_rank_prefill_log.py, test_coresidence_budget_mapping.py -- no assertion weakened, each fixture updated to match the current production shape. fix(sgl-project#499): a server_args field read on both sides of materialize_declarations is TWO values. The sgl-project#89 hibernate manifest identity (model_loader/hibernate.py:_model_identity) is computed at the MATCH inside ServerArgs._handle_load_format BEFORE materialize_declarations(self), and again at the PARK in the worker, long after it. quantization is declared "gguf" at the head of that same handler, so every manifest was written with "gguf" while every subsequent boot compared None -- the identity could never match its own park, and sgl-project#89's fast restore was unreachable on GGUF, the only checkpoint class sgl-project#89 supports. The park kept reporting success and the mismatch silently fell back to a cold load that works, so nothing surfaced. Fixed by computing the identity through resolved_view, the mid-resolution equivalent of the post-materialization fields, so both sides read the same value. HIBERNATE_VERSION stays 2 -- manifests parked before the fix become matchable rather than invalid. Reach read off the metadata: of the nine identity fields only quantization and dtype are declarable at all, so the hazard was confined to those two; a residual (ModelRunner._sm80_dtype_fallback, worker-resolved after the match) is named and left open -- unreachable on sm80+, live only on the sm75 hetero host. FEATURE_CATALOG.md §12 conflict as expected (this branch and the already-merged sgl-project#514 both insert a new paragraph at the same point, right after the sgl-project#501 reclaim-then-decline test line): kept both, sgl-project#514's five paragraphs (two-stage-error, group-agreement, rank-local-verdict, fabricated-identity, value-pinning) first, sgl-project#499's "Resolution-ordering family" appended after -- no content overlap despite the shared "identity" theme (different mechanisms: server_args field declaration ordering vs. NVML->CUDA card index mapping). All 17 sections and every other additive paragraph verified present. Also fixed a missing blank-line paragraph separator this merge introduced, plus three more found from the earlier sgl-project#513/sgl-project#511 §12-header resolutions (cosmetic -- markdown paragraphs need a blank line between them or they render as one block; no content change). Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - The five directly touched/new test files: 47 passed, 4 skipped, 0 failed -- confirms the "7 red -> 0" claim (test_decode_queue_cleanup.py, which failed earlier in this same session's sgl-project#514 verification pass, is now green). - test/registered/unit/distributed/ (full, since test_bar1_strand_is_opt_in.py lives there): 24 failed, 2147 passed, 8 skipped, 721 subtests -- unchanged from the established baseline. - ruff check --select=F401,F821,UP037: clean. - codespell: clean on all touched/added files.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
… pool -- established unreachable, not fixed sgl-project#487 audits whether the stock even-DCP allocator branch (model_runner_kv_cache_mixin.py, the else of the allocator chain -- which inflates both index space and page granularity by dcp_size, assuming a token-sharded pool) can ever run against a draft worker at --draft-kv-layout replicated (the default), whose pool sizing has the opposite geometry (draft_pool_is_replicated) and which the allocator selection never mentions. sgl-project#108 never audited this crossing. Answer established WITHOUT a boot, by enumerating every producer of is_draft_worker=True rather than assuming there is one shape: UNREACHABLE on CUDA. Given dcp_size > 1 the stock branch is taken exactly when rank_tp_ratio is None and not weightless_kv_active(); (1) a speculative draft worker cannot exist in that shape because ServerArgs._handle_dcp_validation refuses dcp_size>1 + speculation on CUDA unless the boot is uneven-weighted DCP or the weightless fast lane; (2) a sgl-project#274 dual-group-lane runner also sets is_draft_worker=True and is NOT speculative, so leg 1 misses it -- closed instead by _lane_server_args_view forcing view.dcp_size = 1. Leg 2 exists only because "a draft pool implies a speculative algorithm" turned out false once the producer set was enumerated, not assumed -- the new falsifier pins that set so a third producer lands as a red test instead of a wrong address. Residual, named rather than fixed: on HIP/ROCm leg 1 does not run (is_hip() short-circuits before the CUDA branch), so the crossing IS admitted there -- left alone, since this fork does not serve ROCm and a desk-guessed change to address computation is exactly the sgl-project#345 right-token/wrong-slot class waiting to happen. Also folds in docs(sgl-project#482): the ARM3_COMPUTE work-matched counter rule and its script updates (scripts/dev/394_s2_proof/). FEATURE_CATALOG.md §12 conflict as expected (this branch and the already-merged sgl-project#510/sgl-project#513/sgl-project#511 all insert a new family paragraph at the same point, right after the §12 header): kept all six additive paragraphs in sequence (Unauthenticated-state-change sgl-project#510, Incomplete-cache-key sgl-project#513, Unreachable-registration sgl-project#81/sgl-project#518, Byte-stride width sgl-project#109/sgl-project#112/sgl-project#512, Tolerance-that-cannot-fail sgl-project#380/sgl-project#511, Reach-before-fix sgl-project#487). All 17 sections and every other previously-preserved additive paragraph (two-axis §1, matrix doctrine §2, sgl-project#514's five families, sgl-project#499's resolution-ordering, MERGE-DUTY owner-sites + SITREP §12, NVENC §13) verified present. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - test_stock_dcp_allocator_reach_487.py alone: 20 passed. - test/registered/unit/distributed/ (full, since model_runner_kv_cache_mixin.py is touched): 24 failed, 2167 passed, 8 skipped, 723 subtests -- same 24 failing test IDs as the established baseline, byte-identical; the +20/+2 deltas are exactly this merge's own new test file. - ruff check --select=F401,F821,UP037: clean. - codespell: clean on all 8 touched/added files.
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
… correct the §1.3 model, and make the sgl-project#431 guard non-blocking Two commits: test(sgl-project#483,sgl-project#459): sgl-project#483 ("check_regressions prices prefill on stale rates") is REFUTED -- check_regressions already prices on the RESOLVED rates (landed with sgl-project#475). What was actually missing was the PIN: the only test calling the function is checkpoint-gated and skips in every hermetic run, so a revert would be invisible. test_check_regressions_pricing_483.py is the hermetic falsifier. Also makes the s12 harness honest: an absent measurement is None, never 0.0 (a spec-off tick has no accept length, and 0.0 silently survives every arithmetic it enters); and any procedural property a verdict rests on (warm-up discarded, draws back-to-back, gap between draws) is recorded as a measured number in the artifact, not taken on the harness's word -- the s12 warm-up draw ran for months and was discarded silently, so a warmed-up point and a cold one wrote the same file. perf(sgl-project#517): names the actual decode-loop BAR1 collectives and corrects the catalog's round model -- a decode round is NOT free of host-path collectives (the belief the old model rested on). Per NEXTN round there are 3 replay boundaries (the draft chain is one captured graph, not one per step) and 5 host-path BAR1 broadcasts, all sgl-project#50 speculative rank-agreement syncs, reaching barlink because a barlink boot never constructs pynccl. Measured cost: the sgl-project#431 blocking status-word read costs -9.22% decode TPS against the same-tree NCCL baseline; removing it recovers +2.68%. Fix: the abort-guard status word is now read asynchronously (non-blocking D2H + cudaEventQuery, returning what an earlier check staged), trading reporting LATENCY for zero stream synchronization per check -- detection is unaffected because ctlStatus is sticky. SGLANG_BARLINK_BAR1_ABORT_MAX_LAG (default 4) bounds the deferral with a can-fail proof; ..._ABORT_DEFER=0 restores the exact pre-sgl-project#517 blocking behavior. NOTE_517_bar1_guard_desk.md has the full model, desk-only (no GPU arm run yet -- ticketed). Two real conflicts, both resolved by combining rather than choosing a side: - FEATURE_CATALOG.md §12 (same insertion-point pattern as every merge today): HEAD's "Verified carve-outs (sgl-project#514)" paragraph kept, this branch's two sgl-project#459 paragraphs ("absent measurement is None" / "procedural property must be in the artifact") appended after. Two OTHER hunks (the sgl-project#483-REFUTED note near §2, the "Guard cost" note near §7) landed with no conflict at all -- neither region had been touched since this branch's base. - test/registered/unit/distributed/_bar1_marker_source.py: both HEAD (via sgl-project#514's earlier edits) and this branch carry DIFFERENT LINE_* pins into barlink.py/barlink_bar1.py, because each was written against its own version of those files. Neither side's numbers are correct once both sets of source edits are combined. Resolved by reading the ACTUAL post-merge source: grepped the real logger.info/logger.warning/raise RuntimeError call-start lines in the merged barlink.py and barlink_bar1.py (2236/2250/3207/810) and verified all four renderers execute correctly against them before committing -- not a textual merge of the two guesses. All 17 catalog sections and every other additive paragraph verified present. Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - The five directly touched/new test files (_bar1_marker_source.py's renderers verified live against the real merged source, plus test_barlink_bar1_abort_431.py, test_barlink_bar1_abort_deferred_517.py, test_s12_harness_459.py, test_check_regressions_pricing_483.py): 67 + 7 passed, 0 failed. - test_s12_log_analyse.py specifically (s12_log_analyse.py is touched): same 4 pre-existing failures, unchanged (fixture-data-dependent, not source-code-dependent). - test/registered/unit/distributed/ (full): 24 failed, 2206 passed, 8 skipped, 723 subtests -- same 24 failing IDs as the established baseline, byte-identical. - ruff check --select=F401,F821,UP037: clean. - codespell: 7 hits, all confirmed pre-existing (a German prose sample string already in scripts/gpu_battery/s12_prefill_kurve.py and s14_decode_punkt.py at this branch's base, untouched by its diff).
efschu
added a commit
to efschu/htsglang
that referenced
this pull request
Aug 3, 2026
… medians, honest server state Three new server-side computed blocks the Monitor tab's landing poll now returns (all server-side so a curl client sees exactly what the page draws, per the runbook's "curl is a complete client" architecture rule): - server_state.py: a four-state discriminator (not_running | starting | running_no_metrics | running_with_metrics) that keys off an explicit API probe, never a metrics-scrape failure alone -- a connection-refused scrape used to be rendered as "no --enable-metrics", which is a different diagnosis than "not running at all". running_no_metrics is only ever reported with a passing API probe. - rate_medians.py: medians behind the rate tiles, windowed over PROCESSING polls only -- idle polls never enter the window, so an idle server still reports the median of what it did while working, not a zero-diluted average. - spill_tiers.py / tier_occupancy.py: one row per spill/offload tier type x place, with absent tiers PRINTED alongside their reason rather than hidden or shown as zero. Two tiers are deliberately never populated: the sgl-project#407 memtier registry (no production consumer, so reading it would print zeros that look measured) and the sgl-project#286 short-term register (no reachable byte ledger in this build) -- both documented as absent-by- construction, not omissions. webui.py and live_metrics.py wire these into the existing landing poll; metrics_reporter.py and metrics_collector.py get the small hooks the new blocks read from. Clean auto-merge on both touched docs (FEATURE_CATALOG.md and rig-runbook.md) -- neither had a conflicting edit on this line since this branch's base (3e3aca2, the sgl-project#510 merge). rig-runbook.md gains new §8.0 documenting the shell-only verification recipe (curl + three python3 one-liners for state/medians/tiers) and the restart requirement: the dashboard serves from one long-lived process, so a change is live only after that process restarts -- done as the merge step below, not mid-session. All 17 catalog sections and all 21 previously-preserved additive paragraphs verified present (two-axis §1, matrix doctrine §2, all six §12 families from sgl-project#510/sgl-project#513/sgl-project#511/sgl-project#487/qwen38-prep, sgl-project#514's five families + value-pinning, sgl-project#499's resolution-ordering, sgl-project#517's two sgl-project#459 paragraphs, MERGE-DUTY owner-sites + SITREP §12, NVENC §13). Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree): - The six directly touched/new test files: 517 passed, 0 failed. - test/registered/unit/planner/ (full, HTSGLANG_TEST_MODEL_DIR set): 2337 passed, 1 skipped, 352 subtests -- no failures. - ruff check --select=F401,F821,UP037: 1 finding (live_metrics.py's LiveSnapshot import), confirmed pre-existing -- already unused in the pre-merge base, only mentioned in a docstring, the line just shifted from sgl-project#522's unrelated insertions above it. - codespell: 1 hit (webui.py:1786 "unparseable"), confirmed pre-existing at the identical line in the pre-merge base, in unrelated SVG chess-piece-parsing code this branch never touches.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Otherwise, the requests will hang forever.