Skip to content

Crash the server when error or OOM happens - #514

Merged
merrymercy merged 2 commits into
mainfrom
fix-crash
Jun 8, 2024
Merged

merrymercy merged 2 commits into
mainfrom
fix-crash

Conversation

@merrymercy

@merrymercy merrymercy commented Jun 8, 2024 •

Copy link
Copy Markdown
Collaborator

Otherwise, the requests will hang forever.

@merrymercy
merrymercy merged commit 91f93f1 into main Jun 8, 2024
@merrymercy
merrymercy deleted the fix-crash branch June 8, 2024 02:22
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
vschandramourya pushed a commit to vschandramourya/sglang that referenced this pull request Feb 3, 2026
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
efschu added a commit to efschu/htsglang that referenced this pull request Aug 3, 2026
…gl-project#506

Closes the four A2 findings from AUDIT_506_unswept_axes.md's state-changing
HTTP endpoint sweep:
- A2-F1: /hibernate no longer accepts an unvalidated hibernate_dir override
  from the request body (path-confinement check, new utils/path_confinement.py)
  and drops the GET method -- POST only.
- A2-F2/F3: the fork's own state-changing routes now carry an explicit
  auth requirement instead of relying on ADMIN_FORCE placement; CORS
  credentials no longer combine with a wildcard origin.
- A2-F4: the registry control plane (registry/launch.py, registry/http_api.py)
  requires auth before accepting an argv-driven engine launch.
- A2-F5: video_enhance/mux.py's ffprobe/ffmpeg source_url gets a scheme
  allowlist and no longer reflects raw subprocess stderr to the caller
  (subprocess_failure() wraps it into a MuxError that omits the detail).
- /v1/files (training/service.py) now guards on the training-enabled flag
  before writing to disk.

Clean auto-merge -- FEATURE_CATALOG.md and server_args.py both merged
without conflict (neither had any other change on this line since the
fceaec3 base). One follow-up fix in the same commit: ruff flagged 2 NEW
UP037 findings in the branch's own new subprocess_failure() (quoted
`"bytes | str | None"` / `"MuxError"` annotations that don't need quoting --
MuxError is defined earlier in the file, no forward-reference issue) --
autofixed with `ruff check --select=UP037 --fix`, confirmed clean after.

Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree):
- test/registered/unit/entrypoints/ + registry/ + training/ + workbench/:
  605 passed, 61 subtests.
- test/registered/video_enhance/ + unit/server_args/: 1186 passed,
  4 skipped, 266 subtests.
- ruff check --select=F401,F821,UP037: clean after the mux.py autofix.
  No ruff format run on planner/* or other listed subtrees (CLAUDE.md
  sgl-project#514-lesson rule) -- not applicable here anyway, no planner files touched.
- codespell clean on all 11 touched/added files.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 3, 2026
…gl-project#506 axis 3)

Incomplete-cache-key family (sgl-project#241/sgl-project#513): a persisted artifact whose CONTENT
depends on a dimension its KEY does not carry is a silent wrong hit, not a
miss. sgl-project#513 closes four instances the sgl-project#506 axis-3 sweep found:
(a) rigmon/card_probe.py wrote a correct key (sorted UUIDs, driver, probe
version) but both readers (planner/solver_api.py,
planner/rig_profile_source.py) took the newest file by mtime instead --
card_probe.matching_cached_probe_json() is now the single reader, a
non-match is a miss.
(b) HiCache page keys carried tp_rank/tp_size, which under uneven TP does
not determine a rank's kv-head count; --rank-tp-ratio/--rank-kv-ratio now
enter compute_model_identity_hash (only when set, so even-TP rigs keep
their persisted pages).
(c) The measured-KV-budget fingerprint omitted attention_backend,
disable_cuda_graph, dtype and enable_hierarchical_cache -- all four change
the post-capture VRAM leftover it records; added on the file's own
only-when-non-default convention.
(d) planner/graphmem.py keyed the NUMBER of capture batch sizes, so two
different --cuda-graph-bs lists of equal length shared an anchor; the key
is now versioned (ANCHOR_KEY_VERSION v2) and carries the list, attention
backend and page size. Two existing graphmem tests had been passing
BECAUSE of the collision -- fixed alongside.

One real conflict, as expected (two branches both add a new paragraph to
the very start of FEATURE_CATALOG.md's §12, right after the header): kept
BOTH additive paragraphs, sgl-project#510's "Unauthenticated-state-change family"
first (already on the line), sgl-project#513's "Incomplete-cache-key family" appended
directly after. uneven_perf.py's fingerprint-fields addition auto-merged
clean (self-contained tail addition, no other change on this line touched
that function). All 17 sections verified intact.

Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree):
- test/registered/unit/planner/ (HTSGLANG_TEST_MODEL_DIR set): 2225 passed,
  1 skipped, 352 subtests (up from 2223 pre-merge, +2 from test_graphmem.py's
  new methods).
- test/registered/unit/mem_cache/ + unit/rigmon/ (excluding
  test_hicache_nixl_storage.py, which needs the external `nixl` package not
  installed in this venv -- environment gap, unrelated to sgl-project#513): 937 failed
  total, but EVERY failure traced to one untouched file,
  test_unified_radix_cache_unittest.py. Verified non-regression by direct
  comparison, not assumption: ran that file alone against both the
  pre-merge base (fceaec3) and the merged tree -- both produce
  byte-identical 851 failed / 16 passed / 648 skipped (diff of the two
  failing-test lists is empty). Root cause is environmental (RuntimeError:
  No accelerator (CUDA/XPU/HPU/NPU/MUSA/MPS) under this hermetic CPU-only
  invocation), not something sgl-project#513 touches or introduces. Every other file
  in the two directories passed clean.
- test/registered/hicache/*.py (top-level, not unit/): needs a live server
  boot + model download (one hits a gated HF repo) -- out of scope for a
  hermetic merge check, unrelated to sgl-project#513, left unexercised.
- ruff check --select=F401,F821,UP037: 5 findings, all confirmed
  pre-existing at fceaec3 and outside sgl-project#513's diff hunks (graphmem.py's
  unused `math` import, three card_probe.py `from_json` UP037 quotes,
  uneven_perf.py's long-standing Dict[str, List[float]] quote). No ruff
  format run on planner/* (CLAUDE.md sgl-project#514-lesson rule).
- codespell clean on all 9 touched/added files.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 3, 2026
…-wrongness sweep

Eight commits, each closing or correcting a sgl-project#505 finding:
- B-01: offload_movement splits the wave-in error path -- a destination
  release that already landed is not a failed retrieval; the item is only
  marked PARKED and its memtier booking only held when the bytes genuinely
  are still at the park target.
- C-01: step 0 of the sgl-project#505-C-01 liveness-timeout task REFUTES its own
  ranking -- TTFT is outside the 90s budget (the streaming response is only
  built after generator.__anext__() returns, so the watchdog does not exist
  during queueing/prefill). The default is deliberately left unchanged;
  what sgl-project#514 pins instead is the ORDERING that makes that true
  (TimeToFirstTokenIsOutsideTheBudgetTest, proven can-fail by removing the
  pre-pull). Same lesson as sgl-project#500-B1: an audit finding is a hypothesis until
  its falsifier is red.
- C-05: CONVENTION_bounding_defaults.md + a value-pinning test convention,
  with the first three defaults pinned against removal/halving.
- A1-01: widens the draft-load completeness check and makes its non-reach
  audible instead of silent.
- A2-04: PD topology returns UNKNOWN instead of fabricating a CUDA->NVML
  identity mapping when it cannot resolve one.
- D3: scrubs the sgl-project#152 GitHub result-share route (was rendering the start
  command's argv verbatim) and binds the posted body to what the user
  actually previewed.
- A2-03/A2-01: the kv-session-offload spill verdict is now rank-uniform
  (a group decision), and a carve-out that used to be assumed is now
  verified against the live state.
- A2-05: the barlink transport fallback is a GROUP decision, not a
  per-rank one. Also PARTIALLY REFUTES the sgl-project#505-A2-05 damage story: the
  byte proof already reduces its verdict group-wide, so the three named
  sites desync into a hang, not a silent split; the genuine split window
  is narrower (a rank-local exception after a transport's last bring-up
  collective) and is what this fix actually closes. The broader desync
  class is untouched, recorded as a separate GPU-window ticket.

Corrects AUDIT_505_silent_wrongness.md and FEATURE_CATALOG.md in place with
the same append-only CORRECTION/REFUTED convention used throughout this
line (blockquotes, nothing deleted). Clean auto-merge, no conflicts in
either file -- all 17 catalog sections and every previously-preserved
additive paragraph (two-axis §1, matrix doctrine §2, sgl-project#510/sgl-project#513/sgl-project#501 §12
families, MERGE-DUTY owner-sites + SITREP §12, NVENC §13) verified present.
test_kvso_reclaim_decline_501.py (sgl-project#501's own test) picks up a 1-line change
from sgl-project#514's kv_session_offload.py work -- sgl-project#514's base already contains
sgl-project#501, so this is sgl-project#514 building on top of it, not a collision.

Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree):
- Targeted run across disaggregation/, the new/changed test files, managers,
  distributed (barlink/gpu_battery), and liveness: 530 passed, 21 subtests,
  1 failure (test_decode_queue_cleanup.py, untouched by sgl-project#514's diff;
  reproduced byte-identically on the pre-merge base 96c7dc5 -- same
  generic retry()-wrapped environmental failure seen throughout this line).
- test/registered/unit/distributed/ (full): 24 failed, 2147 passed,
  8 skipped, 721 subtests -- matches the established 24-failure baseline.
- test/registered/unit/model_loader/ (full, since loader.py/weight_utils.py
  are touched): 32 failed, 229 passed, 9 skipped, 41 subtests on the merged
  tree vs 32 failed, 226 passed on the pre-merge base -- diffed the two
  failing-test-ID lists directly: BYTE-IDENTICAL. The three extra passes
  are sgl-project#514's own new/extended test coverage elsewhere in the directory.
- ruff check --select=F401,F821,UP037: 2 findings (offload_movement.py:206
  "DeviceOps", loader.py's unused _export_static_state import), both
  confirmed pre-existing at 96c7dc5 and outside sgl-project#514's diff hunks. No
  ruff format run (CLAUDE.md sgl-project#514-lesson rule -- ironically the same task
  number).
- codespell: one hit, "unter" in test_gpu_battery_checks.py:1965 (a
  pre-existing German test-fixture string, `"reason": "unter der Marke"`,
  already present at 96c7dc5 and untouched by sgl-project#514's diff -- a
  pre-existing English-only-code violation, not introduced here).
efschu added a commit to efschu/htsglang that referenced this pull request Aug 3, 2026
…pert stride; catch-all probe registration

Three sgl-kernel csrc fixes, each with an off-GPU can-discriminate test
(pure tree change -- wheel NOT built/installed here):

- sgl-project#511 (audit sgl-project#506 axis 4, sgl-project#380 class): the GGUF MXFP4 numeric gate was
  loose enough that an all-zeros or sign-flipped output passed
  (atol=1.5, rtol=3e1 against RMS 5.1e3 outputs). Replaced with a tolerance
  DERIVED from the one physical error source (q8_1 activation rounding),
  plus spread/magnitude preconditions. Every gate in test_gguf_mxfp4_cuda.py
  now has an off-GPU test proving it rejects a zeroed and a sign-flipped
  output, including the refuted baseline executed as its own test.
- sgl-project#512 (audit sgl-project#506 A1-1): the GGUF MoE MMQ kernel's expert-address byte
  stride (moe.cuh) was declared `int` (32-bit) while the call site already
  passed int64_t -- wraps negative once a rank's per-layer expert tensor
  passes 2 GiB (DSV4-Flash Q4_K, 256 experts, first bad local expert 227;
  TP=3 sharding is the only reason this rig never hit it). Widened to
  int64_t through all instantiations.
- sgl-project#518: three GGUF capability probes (ggml_moe_get_block_size,
  ggml_mmvq_kq_tuned, ggml_mxfp4_native) were registered for a DEVICE
  dispatch key but take no tensor argument, so the dispatcher can never
  route them -- every call raised "no tensor arguments" before any kernel
  ran, on every arch. Serving never noticed because gguf.py had grown a
  python mirror around the raise. Fixed with the same keyless m.impl
  pattern apply_token_bitmask_inplace_cuda already uses.
  test_no_tensor_op_dispatch_518.py pins it twice: the dispatcher behaviour
  on throwaway in-process ops (executed, off-GPU, no wheel needed) and a
  ratchet over every TORCH_LIBRARY_FRAGMENT schema.

FEATURE_CATALOG.md §12 conflict as expected (both this branch and the
already-merged sgl-project#510/sgl-project#513 insert a new family paragraph immediately after
the §12 header): kept all three additive paragraphs in sequence
(Unauthenticated-state-change sgl-project#510, Incomplete-cache-key sgl-project#513,
Unreachable-registration sgl-project#81/sgl-project#518, Byte-stride width sgl-project#109/sgl-project#112/sgl-project#512,
Tolerance-that-cannot-fail sgl-project#380/sgl-project#511). All 17 sections and every other
previously-preserved additive paragraph verified present.

Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree, no
wheel rebuild):
- The three sgl-project#511-touched test files directly: 29 passed, 14 skipped (the
  CUDA-gated MXFP4 kernel cases correctly skip without a rebuilt wheel; the
  off-GPU can-discriminate portions all run and pass).
- test/registered/unit/quantization/ (full): 6 failed, 100 passed,
  14 skipped, 27 subtests -- all 6 failures confined to
  test_gguf_dequant_scratch_budget.py, untouched by sgl-project#511's diff.
- test/registered/unit/layers/ (full): 3 failed, 646 passed, 39 skipped,
  995 subtests -- all 3 failures in test_deterministic_fp8_gemm.py /
  test_flashinfer_workspace_zeroing.py, both untouched by sgl-project#511's diff.
- test/registered/unit/model_loader/ (full, since gguf.py is touched):
  32 failed, 229 passed, 9 skipped, 41 subtests. Diffed the FULL
  failing-test-ID list (not just the tail) against the pre-sgl-project#511 tip
  (post-sgl-project#514): BYTE-IDENTICAL, 32 lines each side, empty diff. An earlier
  truncated `tail` comparison looked different due to test-order variance
  in what scrolled past -- re-verified against the complete list before
  trusting it.
- ruff check --select=F401,F821,UP037: clean.
- codespell: clean on all 9 touched/added files including the .cc/.cuh
  sources.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 3, 2026
…ough the declaration overlay

Task sgl-project#465 fixture repair + sgl-project#499 hibernate fix, two commits:

test(sgl-project#465): repairs four stale test fixtures across the line (7 red -> 0):
test_decode_queue_cleanup.py, test_bar1_strand_is_opt_in.py,
test_rank_prefill_log.py, test_coresidence_budget_mapping.py -- no
assertion weakened, each fixture updated to match the current production
shape.

fix(sgl-project#499): a server_args field read on both sides of
materialize_declarations is TWO values. The sgl-project#89 hibernate manifest
identity (model_loader/hibernate.py:_model_identity) is computed at the
MATCH inside ServerArgs._handle_load_format BEFORE
materialize_declarations(self), and again at the PARK in the worker, long
after it. quantization is declared "gguf" at the head of that same
handler, so every manifest was written with "gguf" while every subsequent
boot compared None -- the identity could never match its own park, and
sgl-project#89's fast restore was unreachable on GGUF, the only checkpoint class sgl-project#89
supports. The park kept reporting success and the mismatch silently fell
back to a cold load that works, so nothing surfaced. Fixed by computing
the identity through resolved_view, the mid-resolution equivalent of the
post-materialization fields, so both sides read the same value.
HIBERNATE_VERSION stays 2 -- manifests parked before the fix become
matchable rather than invalid. Reach read off the metadata: of the nine
identity fields only quantization and dtype are declarable at all, so the
hazard was confined to those two; a residual (ModelRunner._sm80_dtype_fallback,
worker-resolved after the match) is named and left open -- unreachable on
sm80+, live only on the sm75 hetero host.

FEATURE_CATALOG.md §12 conflict as expected (this branch and the
already-merged sgl-project#514 both insert a new paragraph at the same point, right
after the sgl-project#501 reclaim-then-decline test line): kept both, sgl-project#514's five
paragraphs (two-stage-error, group-agreement, rank-local-verdict,
fabricated-identity, value-pinning) first, sgl-project#499's "Resolution-ordering
family" appended after -- no content overlap despite the shared "identity"
theme (different mechanisms: server_args field declaration ordering vs.
NVML->CUDA card index mapping). All 17 sections and every other additive
paragraph verified present. Also fixed a missing blank-line paragraph
separator this merge introduced, plus three more found from the earlier
sgl-project#513/sgl-project#511 §12-header resolutions (cosmetic -- markdown paragraphs need a
blank line between them or they render as one block; no content change).

Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree):
- The five directly touched/new test files: 47 passed, 4 skipped, 0
  failed -- confirms the "7 red -> 0" claim (test_decode_queue_cleanup.py,
  which failed earlier in this same session's sgl-project#514 verification pass, is
  now green).
- test/registered/unit/distributed/ (full, since test_bar1_strand_is_opt_in.py
  lives there): 24 failed, 2147 passed, 8 skipped, 721 subtests --
  unchanged from the established baseline.
- ruff check --select=F401,F821,UP037: clean.
- codespell: clean on all touched/added files.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 3, 2026
… pool -- established unreachable, not fixed

sgl-project#487 audits whether the stock even-DCP allocator branch
(model_runner_kv_cache_mixin.py, the else of the allocator chain -- which
inflates both index space and page granularity by dcp_size, assuming a
token-sharded pool) can ever run against a draft worker at
--draft-kv-layout replicated (the default), whose pool sizing has the
opposite geometry (draft_pool_is_replicated) and which the allocator
selection never mentions. sgl-project#108 never audited this crossing.

Answer established WITHOUT a boot, by enumerating every producer of
is_draft_worker=True rather than assuming there is one shape: UNREACHABLE
on CUDA. Given dcp_size > 1 the stock branch is taken exactly when
rank_tp_ratio is None and not weightless_kv_active(); (1) a speculative
draft worker cannot exist in that shape because
ServerArgs._handle_dcp_validation refuses dcp_size>1 + speculation on CUDA
unless the boot is uneven-weighted DCP or the weightless fast lane; (2) a
sgl-project#274 dual-group-lane runner also sets is_draft_worker=True and is NOT
speculative, so leg 1 misses it -- closed instead by
_lane_server_args_view forcing view.dcp_size = 1. Leg 2 exists only
because "a draft pool implies a speculative algorithm" turned out false
once the producer set was enumerated, not assumed -- the new falsifier
pins that set so a third producer lands as a red test instead of a wrong
address. Residual, named rather than fixed: on HIP/ROCm leg 1 does not run
(is_hip() short-circuits before the CUDA branch), so the crossing IS
admitted there -- left alone, since this fork does not serve ROCm and a
desk-guessed change to address computation is exactly the sgl-project#345
right-token/wrong-slot class waiting to happen.

Also folds in docs(sgl-project#482): the ARM3_COMPUTE work-matched counter rule and
its script updates (scripts/dev/394_s2_proof/).

FEATURE_CATALOG.md §12 conflict as expected (this branch and the
already-merged sgl-project#510/sgl-project#513/sgl-project#511 all insert a new family paragraph at the
same point, right after the §12 header): kept all six additive paragraphs
in sequence (Unauthenticated-state-change sgl-project#510, Incomplete-cache-key sgl-project#513,
Unreachable-registration sgl-project#81/sgl-project#518, Byte-stride width sgl-project#109/sgl-project#112/sgl-project#512,
Tolerance-that-cannot-fail sgl-project#380/sgl-project#511, Reach-before-fix sgl-project#487). All 17
sections and every other previously-preserved additive paragraph
(two-axis §1, matrix doctrine §2, sgl-project#514's five families, sgl-project#499's
resolution-ordering, MERGE-DUTY owner-sites + SITREP §12, NVENC §13)
verified present.

Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree):
- test_stock_dcp_allocator_reach_487.py alone: 20 passed.
- test/registered/unit/distributed/ (full, since
  model_runner_kv_cache_mixin.py is touched): 24 failed, 2167 passed,
  8 skipped, 723 subtests -- same 24 failing test IDs as the established
  baseline, byte-identical; the +20/+2 deltas are exactly this merge's own
  new test file.
- ruff check --select=F401,F821,UP037: clean.
- codespell: clean on all 8 touched/added files.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 3, 2026
… correct the §1.3 model, and make the sgl-project#431 guard non-blocking

Two commits:

test(sgl-project#483,sgl-project#459): sgl-project#483 ("check_regressions prices prefill on stale rates")
is REFUTED -- check_regressions already prices on the RESOLVED rates
(landed with sgl-project#475). What was actually missing was the PIN: the only test
calling the function is checkpoint-gated and skips in every hermetic run,
so a revert would be invisible. test_check_regressions_pricing_483.py is
the hermetic falsifier. Also makes the s12 harness honest: an absent
measurement is None, never 0.0 (a spec-off tick has no accept length, and
0.0 silently survives every arithmetic it enters); and any procedural
property a verdict rests on (warm-up discarded, draws back-to-back, gap
between draws) is recorded as a measured number in the artifact, not taken
on the harness's word -- the s12 warm-up draw ran for months and was
discarded silently, so a warmed-up point and a cold one wrote the same
file.

perf(sgl-project#517): names the actual decode-loop BAR1 collectives and corrects the
catalog's round model -- a decode round is NOT free of host-path
collectives (the belief the old model rested on). Per NEXTN round there
are 3 replay boundaries (the draft chain is one captured graph, not one
per step) and 5 host-path BAR1 broadcasts, all sgl-project#50 speculative
rank-agreement syncs, reaching barlink because a barlink boot never
constructs pynccl. Measured cost: the sgl-project#431 blocking status-word read costs
-9.22% decode TPS against the same-tree NCCL baseline; removing it
recovers +2.68%. Fix: the abort-guard status word is now read
asynchronously (non-blocking D2H + cudaEventQuery, returning what an
earlier check staged), trading reporting LATENCY for zero stream
synchronization per check -- detection is unaffected because ctlStatus is
sticky. SGLANG_BARLINK_BAR1_ABORT_MAX_LAG (default 4) bounds the
deferral with a can-fail proof; ..._ABORT_DEFER=0 restores the exact
pre-sgl-project#517 blocking behavior. NOTE_517_bar1_guard_desk.md has the full
model, desk-only (no GPU arm run yet -- ticketed).

Two real conflicts, both resolved by combining rather than choosing a
side:
- FEATURE_CATALOG.md §12 (same insertion-point pattern as every merge
  today): HEAD's "Verified carve-outs (sgl-project#514)" paragraph kept, this
  branch's two sgl-project#459 paragraphs ("absent measurement is None" /
  "procedural property must be in the artifact") appended after. Two
  OTHER hunks (the sgl-project#483-REFUTED note near §2, the "Guard cost" note near
  §7) landed with no conflict at all -- neither region had been touched
  since this branch's base.
- test/registered/unit/distributed/_bar1_marker_source.py: both HEAD (via
  sgl-project#514's earlier edits) and this branch carry DIFFERENT LINE_* pins into
  barlink.py/barlink_bar1.py, because each was written against its own
  version of those files. Neither side's numbers are correct once both
  sets of source edits are combined. Resolved by reading the ACTUAL
  post-merge source: grepped the real logger.info/logger.warning/raise
  RuntimeError call-start lines in the merged barlink.py and
  barlink_bar1.py (2236/2250/3207/810) and verified all four renderers
  execute correctly against them before committing -- not a textual
  merge of the two guesses.

All 17 catalog sections and every other additive paragraph verified
present.

Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree):
- The five directly touched/new test files (_bar1_marker_source.py's
  renderers verified live against the real merged source, plus
  test_barlink_bar1_abort_431.py, test_barlink_bar1_abort_deferred_517.py,
  test_s12_harness_459.py, test_check_regressions_pricing_483.py): 67 + 7
  passed, 0 failed.
- test_s12_log_analyse.py specifically (s12_log_analyse.py is touched):
  same 4 pre-existing failures, unchanged (fixture-data-dependent, not
  source-code-dependent).
- test/registered/unit/distributed/ (full): 24 failed, 2206 passed,
  8 skipped, 723 subtests -- same 24 failing IDs as the established
  baseline, byte-identical.
- ruff check --select=F401,F821,UP037: clean.
- codespell: 7 hits, all confirmed pre-existing (a German prose sample
  string already in scripts/gpu_battery/s12_prefill_kurve.py and
  s14_decode_punkt.py at this branch's base, untouched by its diff).
efschu added a commit to efschu/htsglang that referenced this pull request Aug 3, 2026
… medians, honest server state

Three new server-side computed blocks the Monitor tab's landing poll now
returns (all server-side so a curl client sees exactly what the page
draws, per the runbook's "curl is a complete client" architecture rule):

- server_state.py: a four-state discriminator (not_running | starting |
  running_no_metrics | running_with_metrics) that keys off an explicit API
  probe, never a metrics-scrape failure alone -- a connection-refused
  scrape used to be rendered as "no --enable-metrics", which is a
  different diagnosis than "not running at all". running_no_metrics is
  only ever reported with a passing API probe.
- rate_medians.py: medians behind the rate tiles, windowed over
  PROCESSING polls only -- idle polls never enter the window, so an idle
  server still reports the median of what it did while working, not a
  zero-diluted average.
- spill_tiers.py / tier_occupancy.py: one row per spill/offload tier type
  x place, with absent tiers PRINTED alongside their reason rather than
  hidden or shown as zero. Two tiers are deliberately never populated:
  the sgl-project#407 memtier registry (no production consumer, so reading it would
  print zeros that look measured) and the sgl-project#286 short-term register (no
  reachable byte ledger in this build) -- both documented as absent-by-
  construction, not omissions.

webui.py and live_metrics.py wire these into the existing landing poll;
metrics_reporter.py and metrics_collector.py get the small hooks the new
blocks read from.

Clean auto-merge on both touched docs (FEATURE_CATALOG.md and
rig-runbook.md) -- neither had a conflicting edit on this line since this
branch's base (3e3aca2, the sgl-project#510 merge). rig-runbook.md gains new
§8.0 documenting the shell-only verification recipe (curl + three python3
one-liners for state/medians/tiers) and the restart requirement: the
dashboard serves from one long-lived process, so a change is live only
after that process restarts -- done as the merge step below, not
mid-session. All 17 catalog sections and all 21 previously-preserved
additive paragraphs verified present (two-axis §1, matrix doctrine §2,
all six §12 families from sgl-project#510/sgl-project#513/sgl-project#511/sgl-project#487/qwen38-prep, sgl-project#514's five
families + value-pinning, sgl-project#499's resolution-ordering, sgl-project#517's two sgl-project#459
paragraphs, MERGE-DUTY owner-sites + SITREP §12, NVENC §13).

Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree):
- The six directly touched/new test files: 517 passed, 0 failed.
- test/registered/unit/planner/ (full, HTSGLANG_TEST_MODEL_DIR set):
  2337 passed, 1 skipped, 352 subtests -- no failures.
- ruff check --select=F401,F821,UP037: 1 finding
  (live_metrics.py's LiveSnapshot import), confirmed pre-existing --
  already unused in the pre-merge base, only mentioned in a docstring,
  the line just shifted from sgl-project#522's unrelated insertions above it.
- codespell: 1 hit (webui.py:1786 "unparseable"), confirmed pre-existing
  at the identical line in the pre-merge base, in unrelated SVG
  chess-piece-parsing code this branch never touches.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant