Skip to content

Use model loader from vllm - #459

Merged
merrymercy merged 9 commits into
mainfrom
grok
May 21, 2024
Merged

merrymercy merged 9 commits into
mainfrom
grok

Conversation

@merrymercy

Copy link
Copy Markdown
Collaborator

No description provided.

@merrymercy
merrymercy merged commit 19d2135 into main May 21, 2024
@merrymercy
merrymercy deleted the grok branch May 21, 2024 16:14
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
vschandramourya pushed a commit to vschandramourya/sglang that referenced this pull request Feb 3, 2026
…ject#459)

* Copy OSS code from d8faf2f on 20251204

* fix TRTLLMMLABackend and remove trtllm_mla_tgl_backend.py

* fix lint

* add IS_BLACKWELL=1 to b200 test to have longer timeout

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
EdwardXuy pushed a commit to shun8686/sglang that referenced this pull request Apr 30, 2026
efschu added a commit to efschu/htsglang that referenced this pull request Aug 3, 2026
…ockstep max (two all-reduces per layer), barrier skew reported per candidate; zero-parameter skew prediction 27.6 vs 27.9 ms/1k measured; FP8 ladder byte-identical, INT8 honestly has no prefill lever on this rig; floor-explosion root = harness clock ramp (-> sgl-project#459/sgl-project#483)
efschu added a commit to efschu/htsglang that referenced this pull request Aug 3, 2026
… correct the §1.3 model, and make the sgl-project#431 guard non-blocking

Two commits:

test(sgl-project#483,sgl-project#459): sgl-project#483 ("check_regressions prices prefill on stale rates")
is REFUTED -- check_regressions already prices on the RESOLVED rates
(landed with sgl-project#475). What was actually missing was the PIN: the only test
calling the function is checkpoint-gated and skips in every hermetic run,
so a revert would be invisible. test_check_regressions_pricing_483.py is
the hermetic falsifier. Also makes the s12 harness honest: an absent
measurement is None, never 0.0 (a spec-off tick has no accept length, and
0.0 silently survives every arithmetic it enters); and any procedural
property a verdict rests on (warm-up discarded, draws back-to-back, gap
between draws) is recorded as a measured number in the artifact, not taken
on the harness's word -- the s12 warm-up draw ran for months and was
discarded silently, so a warmed-up point and a cold one wrote the same
file.

perf(sgl-project#517): names the actual decode-loop BAR1 collectives and corrects the
catalog's round model -- a decode round is NOT free of host-path
collectives (the belief the old model rested on). Per NEXTN round there
are 3 replay boundaries (the draft chain is one captured graph, not one
per step) and 5 host-path BAR1 broadcasts, all sgl-project#50 speculative
rank-agreement syncs, reaching barlink because a barlink boot never
constructs pynccl. Measured cost: the sgl-project#431 blocking status-word read costs
-9.22% decode TPS against the same-tree NCCL baseline; removing it
recovers +2.68%. Fix: the abort-guard status word is now read
asynchronously (non-blocking D2H + cudaEventQuery, returning what an
earlier check staged), trading reporting LATENCY for zero stream
synchronization per check -- detection is unaffected because ctlStatus is
sticky. SGLANG_BARLINK_BAR1_ABORT_MAX_LAG (default 4) bounds the
deferral with a can-fail proof; ..._ABORT_DEFER=0 restores the exact
pre-sgl-project#517 blocking behavior. NOTE_517_bar1_guard_desk.md has the full
model, desk-only (no GPU arm run yet -- ticketed).

Two real conflicts, both resolved by combining rather than choosing a
side:
- FEATURE_CATALOG.md §12 (same insertion-point pattern as every merge
  today): HEAD's "Verified carve-outs (sgl-project#514)" paragraph kept, this
  branch's two sgl-project#459 paragraphs ("absent measurement is None" /
  "procedural property must be in the artifact") appended after. Two
  OTHER hunks (the sgl-project#483-REFUTED note near §2, the "Guard cost" note near
  §7) landed with no conflict at all -- neither region had been touched
  since this branch's base.
- test/registered/unit/distributed/_bar1_marker_source.py: both HEAD (via
  sgl-project#514's earlier edits) and this branch carry DIFFERENT LINE_* pins into
  barlink.py/barlink_bar1.py, because each was written against its own
  version of those files. Neither side's numbers are correct once both
  sets of source edits are combined. Resolved by reading the ACTUAL
  post-merge source: grepped the real logger.info/logger.warning/raise
  RuntimeError call-start lines in the merged barlink.py and
  barlink_bar1.py (2236/2250/3207/810) and verified all four renderers
  execute correctly against them before committing -- not a textual
  merge of the two guesses.

All 17 catalog sections and every other additive paragraph verified
present.

Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree):
- The five directly touched/new test files (_bar1_marker_source.py's
  renderers verified live against the real merged source, plus
  test_barlink_bar1_abort_431.py, test_barlink_bar1_abort_deferred_517.py,
  test_s12_harness_459.py, test_check_regressions_pricing_483.py): 67 + 7
  passed, 0 failed.
- test_s12_log_analyse.py specifically (s12_log_analyse.py is touched):
  same 4 pre-existing failures, unchanged (fixture-data-dependent, not
  source-code-dependent).
- test/registered/unit/distributed/ (full): 24 failed, 2206 passed,
  8 skipped, 723 subtests -- same 24 failing IDs as the established
  baseline, byte-identical.
- ruff check --select=F401,F821,UP037: clean.
- codespell: 7 hits, all confirmed pre-existing (a German prose sample
  string already in scripts/gpu_battery/s12_prefill_kurve.py and
  s14_decode_punkt.py at this branch's base, untouched by its diff).
efschu added a commit to efschu/htsglang that referenced this pull request Aug 3, 2026
… medians, honest server state

Three new server-side computed blocks the Monitor tab's landing poll now
returns (all server-side so a curl client sees exactly what the page
draws, per the runbook's "curl is a complete client" architecture rule):

- server_state.py: a four-state discriminator (not_running | starting |
  running_no_metrics | running_with_metrics) that keys off an explicit API
  probe, never a metrics-scrape failure alone -- a connection-refused
  scrape used to be rendered as "no --enable-metrics", which is a
  different diagnosis than "not running at all". running_no_metrics is
  only ever reported with a passing API probe.
- rate_medians.py: medians behind the rate tiles, windowed over
  PROCESSING polls only -- idle polls never enter the window, so an idle
  server still reports the median of what it did while working, not a
  zero-diluted average.
- spill_tiers.py / tier_occupancy.py: one row per spill/offload tier type
  x place, with absent tiers PRINTED alongside their reason rather than
  hidden or shown as zero. Two tiers are deliberately never populated:
  the sgl-project#407 memtier registry (no production consumer, so reading it would
  print zeros that look measured) and the sgl-project#286 short-term register (no
  reachable byte ledger in this build) -- both documented as absent-by-
  construction, not omissions.

webui.py and live_metrics.py wire these into the existing landing poll;
metrics_reporter.py and metrics_collector.py get the small hooks the new
blocks read from.

Clean auto-merge on both touched docs (FEATURE_CATALOG.md and
rig-runbook.md) -- neither had a conflicting edit on this line since this
branch's base (3e3aca2, the sgl-project#510 merge). rig-runbook.md gains new
§8.0 documenting the shell-only verification recipe (curl + three python3
one-liners for state/medians/tiers) and the restart requirement: the
dashboard serves from one long-lived process, so a change is live only
after that process restarts -- done as the merge step below, not
mid-session. All 17 catalog sections and all 21 previously-preserved
additive paragraphs verified present (two-axis §1, matrix doctrine §2,
all six §12 families from sgl-project#510/sgl-project#513/sgl-project#511/sgl-project#487/qwen38-prep, sgl-project#514's five
families + value-pinning, sgl-project#499's resolution-ordering, sgl-project#517's two sgl-project#459
paragraphs, MERGE-DUTY owner-sites + SITREP §12, NVENC §13).

Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree):
- The six directly touched/new test files: 517 passed, 0 failed.
- test/registered/unit/planner/ (full, HTSGLANG_TEST_MODEL_DIR set):
  2337 passed, 1 skipped, 352 subtests -- no failures.
- ruff check --select=F401,F821,UP037: 1 finding
  (live_metrics.py's LiveSnapshot import), confirmed pre-existing --
  already unused in the pre-merge base, only mentioned in a docstring,
  the line just shifted from sgl-project#522's unrelated insertions above it.
- codespell: 1 hit (webui.py:1786 "unparseable"), confirmed pre-existing
  at the identical line in the pre-merge base, in unrelated SVG
  chess-piece-parsing code this branch never touches.
efschu added a commit to efschu/htsglang that referenced this pull request Aug 3, 2026
…-cost probe, the F2 instrument

sgl-project#462 (the breakable MoE-offload CUDA-graph route) was already merged onto
this line long ago; this pulls in two new commits added on top since,
building the sgl-project#494 F2 measurement instrument that ticket named as missing.

srt/utils/break_cost_clock.py prices ONE CUDA-graph crossing of a
breakable capture (segment_end -> eager slot -> segment_start) with a CUDA
event pair around every segment and every break slot in
BreakableCUDAGraph.replay(). Per crossing: gap_in_ms / slot_ms / gap_out_ms
(device wait, device compute, device wait) plus host_ms split into the
four F2 terms (rendezvous, planning, publish, fetch), bracketed via
break_cost_phase() in MoEExpertOffloadCache.prepare_breakable; per round:
compute_ms / wait_ms / span_ms / residual_ms, per rank, as JSONL. Read via
SGLANG_BREAK_COST_DEFER_ROUNDS (default 2) rounds late through query()
only, so the read never synchronises the round it measures.
SGLANG_BREAK_COST_PROBE=1, OFF by default and byte-neutral off (the
disabled replay() loop is the original one statement-for-statement -- no
event created, none recorded, nothing allocated per round, pinned by a
call-count spy). Route-agnostic: any eager_on_graph break point is priced,
not only the MoE one. Reader: scripts/dev/494_break_cost/summarise.py.
21 hermetic tests, three executed can-fail arms. No GPU number exists yet
-- the probe has never run on a card; it is the instrument F2 was missing,
not a result.

FEATURE_CATALOG.md §16 conflict as expected: this branch's own base
predates every merge from today (sgl-project#493/sgl-project#505/sgl-project#459/etc.), so its diff carried
the OLD short §16 base prose alongside its new "Break-cost probe"
paragraph. Kept HEAD's full precise §16 (all of today's corrections) and
appended the Break-cost-probe paragraph after it, discarding only the
superseded old base text the branch's own diff had inherited. §3's small
in-place edit ("its instrument now exists (sgl-project#494, §16 break-cost probe)")
landed with no conflict at all -- untouched by anything else on the line.
All 17 sections and every other additive paragraph verified present.

Tests (CUDA_VISIBLE_DEVICES=99, PYTHONPATH pinned to this worktree; test
tree is tests/moe_offload/, not test/registered/):
- test_break_cost_probe_494.py + test_breakable_route_462.py: 58 passed
  (21 + 37, matching the catalog's own count for each file).
- tests/moe_offload/ (full): 2 failed, 455 passed, 34 skipped -- both
  failures in test_gguf_moe_offload.py, a file this branch's diff never
  touches. Bisected across today's entire merge history AND the prior
  session's tips back to 9fab358 (before sgl-project#500's audit branch base):
  the same 2 failures reproduce identically at every single point,
  including before this session's work began. Confirmed NOT a new
  regression from sgl-project#494/sgl-project#462 or anything merged today -- a long-standing
  pre-existing issue, out of scope for this merge.
- ruff check --select=F401,F821,UP037: 18 findings in expert_offload.py
  and break_cost_clock.py (including two F821 undefined-name hits),
  ALL confirmed pre-existing at cc5a92e and outside this branch's
  three diff hunks (import addition + two break_cost_phase() wraps).
- codespell: clean on all 6 touched/added files.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant