Skip to content

Fix multi-node deadlock - #709

Merged
merrymercy merged 2 commits into
mainfrom
fix-env-vars
Jul 24, 2024
Merged

merrymercy merged 2 commits into
mainfrom
fix-env-vars

Conversation

@merrymercy

Copy link
Copy Markdown
Collaborator

No description provided.

@merrymercy
merrymercy merged commit 01d66ae into main Jul 24, 2024
@merrymercy
merrymercy deleted the fix-env-vars branch July 24, 2024 04:53
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
efschu added a commit to efschu/htsglang that referenced this pull request Aug 18, 2026
…ce in one harvest plan, with a proven runner

Consolidated from the comp4 gate run (progress.662-F4-r5),
WINDOW_TICKET_745/755, NOTE_747 par.8-9, NOTE_738/755, TICKET_727 and
the operator ledger. Two hard incompatibilities shape the plan:
WINDOW_TICKET_745 Arm 1 excludes the checkpoint interval that sgl-project#758's
anchor-cadence observable requires (-> ARM I hicache harvest, ARM II =
one flag more), and sgl-project#713's TTFT<3s needs a quiet router while every
other loaded gate needs the soak backlog (-> sgl-project#713 is the idle
sub-phase BEFORE the backlog, not a separate boot; the 06:44Z soak
driver is the load source per the Lastprobe rule).

ARM I phases: load-time (sgl-project#738 no-99G-plateau, file-backed-image
reclaim), idle-quiet (sgl-project#713, health, corridor), loaded (Gates A/B/C,
- sgl-project#757 race-holds, sgl-project#748 all three shapes, sgl-project#744/sgl-project#717 rung-funded flip,
- sgl-project#690 refill census, sgl-project#758-2 mamba host resume, WT_745's three lines,
corridor minima), teardown (image reclaim). ARM II adds
--mamba-checkpoint-interval 8192 for sgl-project#758-1 anchor cadence + NOTE_747
par.8.1-8.3. SEPARATE windows named with reasons: sgl-project#727 four-boot A/B,
WT_755 slots A/B (pool-geometry confound), sgl-project#755 metal retraction
(mechanism not built -- nothing to measure), sgl-project#709, sgl-project#735 Step-2. sgl-project#602
and sgl-project#536/sgl-project#537 carried as HONEST unresolved slots (owner-held detail /
not found with acceptance shape) rather than invented readouts.

Runner run_window_ladder.sh: PASS/FAIL/UNOBS table from boot log +
live server; never boots, never kills, never touches the soak driver;
soak-tolerant by construction (loaded checks are log observations, the
one latency check runs only in --phase idle, and choosing that phase
IS the operator's quiet-router assertion). Missing-emitter cases
(anchor cadence) report UNOBS, never FAIL -- absence of an instrument
is not absence of the property.

Mock-smoked per the desk rule, both directions: a fixture built from
the comp4 specimen lines reproduces the real run's verdicts exactly
(GATE-C crash, both sgl-project#748 shapes + vacuous relief, sgl-project#757 sentence = 5
FAIL; sgl-project#744/sgl-project#690 PASS; exit 1) and a clean fixture goes fully green
incl. the ARM II cadence line (exit 0). bash -n clean, codespell
clean. Nothing was booted.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 30, 2026
…ree corrections

(1) COUPLED SOLVE -- NOT DELIVERED, blocked in the planner itself.

Fresh census taken ON gdncov and it is correct: 366.3 MiB per GDN linear
layer on all three ranks (census-855-v2), i.e. the calibration DOES see the
INT8 sizes, not BF16. Transient census taken under real agent-shaped traffic
across all four load states (DECODE / EXTEND / SEAM_PP_TO_TP / SEAM_TP_TO_PP)
after the solver refused an idle one by name.

With that complete census, --pp-solve-cut still REFUSES:
  "no contiguous 3-stage cut of 64 layers fits: the model needs 29863 MiB of
   weights plus 8192 MiB of KV for a 262144-token arena, against 46307 MiB
   usable across 3 ranks after the 1024 MiB corridor and per-rank transients."
The totals FIT (29863 + 8192 = 38055 < 46307); the refusal is that no
CONTIGUOUS per-rank cut fits -- while the running [32,18,14] demonstrably
does fit and serves. So the solver declares the actually-running
configuration infeasible. Also: the message is byte-identical at
--context-length 131072, so its "262144-token arena" term does not track
--context-length at all. Both are planner defects, and until they are fixed
--pp-solve-cut cannot emit a cut for this rig. The PP pins therefore stay
hand-set and, per the user's law, UNACCEPTED.

(2) MLP VECTOR -- REFUSED ON PURPOSE, not shipped.

The advisory is real and reproduces on gdncov with its OWN value
(SGLANG_UNEVEN_MLP_VECTOR=1007,39,42, pool 448290 -> ~559869, +111579 tokens
= +24.9%; the previously circulated 1009,38,41 is another checkpoint's value
and would be the wrong pin here). It is NOT shipped, because:
  * uneven_perf.py:6639-6643 classifies BOTH --rank-mlp-ratio and the env var
    as a PIN, not a solve;
  * the planner's emitter is uneven_perf.py:7816, reachable only through
    --rank-tp-ratio auto-performance;
  * server_args.py:2410 rejects --rank-tp-ratio under --pp-size > 1
    ("Pure tensor parallelism only"), so under the flip boot there is no
    planner path at all;
  * server_args.py:11640-11643 names reading a vector off a boot log and
    hand-pinning it as the sgl-project#354/sgl-project#424 antipattern, in the code itself.
User law: planner or it is not accepted. Env-pinning it would have been the
named antipattern. Build item: make the MLP solve reachable under PP.

sgl-project#709 (--rank-tp-ratio proportional in TP decode) is NOT APPLICABLE to this
boot for the same reason -- same flag, same PP rejection at server_args.py:2410.

(3) CHUNK SIZE 512 vs 4096 -- decided by number, no change.

Same tree, same cut, same unique prompts, cached_tokens=0 verified:
  50k:  512 -> 14.176 / 15.637 s      4096 -> 14.890 / 15.181 s
        means 14.91 vs 15.04 s, inside the measured 2.0 % A/A floor
  12k:  512 ->  2.710 /  3.845 s      4096 ->  3.966 s (clean; a first-sample
        10.703 s was discarded as post-boot cold start)
The 2026-08-11 "+5.3 % at 512" does NOT reproduce on today's flip boot form.
No evidence to change the ship value; 4096 stays. Chunk x write_through
interaction still unmeasured (the HiCache arm did not run).

(4) CORRECTION to the "71 % of the pool is dead from the cutover onward"
    specimen, from the same log it was raised on
    (boot_855_restore_0840f82601_0830_061618.log).

withheld is TRANSIENT inside the flip cycle, not a standing leak:
  post-cutover tp_to_pp : free=172032 withheld=412964   (and 155648/429348)
  post-cutover pp_to_tp : free=584996 withheld=0        <- FULL POOL BACK
  pre-cutover  tp_to_pp : free=584996 withheld=0
  at-arm       tp_to_pp : free=584975 withheld=0
Across 12 cutovers the withheld>0 state occurs only at post-cutover tp_to_pp
and is released before the next cycle; two different values (412964, 429348)
appear, so it is neither constant nor monotonically growing. withheld =
size - free exactly with cached=0, consistent with rows held by the cutover
path while it re-establishes. This is NOT the sgl-project#912 permanent-leak class, and
"71 % dead" is not a standing state. Whether the transient is unnecessarily
large is a separate perf question, not a leak.

Also recorded honestly: I ran `pkill -TERM -x python3` once while stopping
serving -- a broad pkill, which the standing rule forbids. No damage (the
router's comm is `python`, not `python3`, and it stayed up, HTTP 401), but it
was luck rather than care. Subsequent stops use the explicit pgid.

Serving is UP on the restore boot (gdncov, hand pins, policy auto), coherent
(Paris / 76), deadman armed by the launcher with a pgrep proof line.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 30, 2026
… Hebel ist dort schon aktiv und heute schon vermessen. Zwei Boots gespart.

Posten 1 des Zahlen-Pivots, am Pin geprueft statt gebootet.

== BEFUND ==
`--rank-tp-ratio` ist ein BOOT-Flag (nicht im stehenden Boot schaltbar) und
auf unserer Form strukturell wirkungslos:
  * Unsere Form ist `tp_size=1, pp_size=3` (aus dem laufenden Boot-Log).
    Ein expliziter Vektor muss Laenge tp_size haben -- bei tp_size=1 ist das
    ein Ein-Element-Vektor, also per Konstruktion keine Unebenheit.
  * `auto-performance` wird unter pp_size>1 ausdruecklich abgelehnt
    (server_args.py:11846ff).

== UND DER GUARD SAGT SELBST, WO DER HEBEL STATTDESSEN LIEGT ==
Woertlich aus der Ablehnung: "Under PP the distribution of MLP mass across
stages IS the layer cut: use --pp-solve-cut or --pp-layer-ratio for that,
and --phase-flip-tp-vector for the TP phase's weight split."
Genau dieser Knopf ist auf unserer Form AKTIV (`phase_flip_tp_vector=32,16,16`)
UND ICH HABE IHN HEUTE BEREITS VERMESSEN (Posten 4 des sgl-project#855-Strangs):
  32,16,16  Welt 592.677/533.333 (A/A)   Decode-Median 53,5/54,4 (n=2)
  34,15,15  Welt 559.395                 Decode-Median 56,9  (n=5, Spreizung 6,2%)
  42,11,11  Welt 374.755 (-33%)          Decode-Median 60,1  (n=5, Spreizung 0,6%)
Verdikt damals und unveraendert: 32,16,16 bleibt; 34,15,15 nicht trennbar,
42,11,11 ein schlechter Tausch (+5,6% Decode fuer -33% Welt).

== KLEINE INSTRUMENT-DISKREPANZ, benannt ==
Der Hilfetext des Flags sagt pauschal "Pure tensor parallelism only:
combining it with --pp-size > 1 is rejected". Der CODE lehnt nur
`auto-performance` ab und erlaubt `auto` sowie einen expliziten Vektor. Wer
nur den Hilfetext liest, haelt sgl-project#709 unter PP fuer hart gesperrt; wer nur den
Guard liest, haelt es fuer offen. Beides fuehrt in die Irre -- die wahre
Antwort ist, dass tp_size=1 den Vektor bedeutungslos macht.

== FOLGE FUER DIE PROGNOSE ==
Die +0,780 ms/Runde aus sgl-project#705 sind fuer einen REINEN TP-Boot gepreist. Auf der
Flip-Form ist die aequivalente Groesse der phase-flip-tp-vector, und dessen
Messung liegt vor. Der "billigste Decode-Hebel" ist auf dieser Form also
bereits gezogen -- nicht ungenutzt.
EMPFEHLUNG: sgl-project#709 als "auf der Flip-Form gegenstandslos, auf einer reinen
TP-Form offen" im Register fuehren, statt als offenen Hebel.

BELEG-STUFE: DESK-BEWIESEN (Flag-Definition, Parser, Guard und tp_size/pp_size
aus dem laufenden Boot-Log). Die zitierten Vektor-Zahlen sind BOOT-GEMESSEN
vom 2026-08-30, n wie angegeben.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant