Skip to content

Fix RuntimeEndpoint - #279

Merged
merrymercy merged 2 commits into
mainfrom
fix-endpoint
Mar 11, 2024
Merged

merrymercy merged 2 commits into
mainfrom
fix-endpoint

Conversation

@merrymercy

Copy link
Copy Markdown
Collaborator

Fix bugs after #230

@merrymercy merrymercy changed the title Fix endpoint Fix RuntimeEndpoint Mar 11, 2024
@merrymercy
merrymercy merged commit 13662fd into main Mar 11, 2024
@merrymercy
merrymercy deleted the fix-endpoint branch March 11, 2024 12:24
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
Garrybest pushed a commit to Garrybest/sglang that referenced this pull request Jan 9, 2026
vschandramourya pushed a commit to vschandramourya/sglang that referenced this pull request Feb 3, 2026
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
liuxianglong17 pushed a commit to shun8686/sglang that referenced this pull request Apr 8, 2026
efschu added a commit to efschu/htsglang that referenced this pull request Jul 28, 2026
…weiter Comm-Arbiter — lane-keyed, prioritaetsbewusst, graph-safe
efschu added a commit to efschu/htsglang that referenced this pull request Jul 29, 2026
…spitze auf NEXTN, konfigurierbar (Sensor aus D1/sgl-project#279, Aktor Plan-Flip, ein Politik-Objekt mit Aktoren Algo/K/topk, sgl-project#141 verwandt)
efschu added a commit to efschu/htsglang that referenced this pull request Jul 29, 2026
…register.py — generisches VRAM-Posten-Tiering (Klassen graph_rungs/drafter_heads/lane_workspaces/cold_lane/experts, Regler resident|ram|auto + fraction, Presets --lane-offload-profile, phase_mask + Zeitkonstanten-Stufen wave|phase|turn, on_phase_boundary-No-Op-Planer mit injizierbarer Overlap-Kostenfunktion, Latenz-Term-API fuer sgl-project#279); Invarianten (Hysterese/Hot-Ablehnung/Prio-Schutz/Fraction-Kappe) in park() und Planer erzwungen; Adapter hinter SGLANG_OFFLOAD_REGISTER=1, Default-Pfad unveraendert; Audit-Tabelle Experten-Maschinerie in INTEGRATION_R3_VALIDATION; Tests 42 neu + 152 Concurrency gruen, ruff/codespell sauber
efschu added a commit to efschu/htsglang that referenced this pull request Jul 29, 2026
…t): htccl_path_dispatcher.py (PathProfile affine ms(bytes) + Kapazitaet + effektive Apertur + measured|placeholder je Eintrag; decide() je Klasse/Groesse/Lane/Prio mit Saettigungs-Ueberlauf, Latch je log2-Band, Flips nur an Runden-/Capture-Grenzen) + htccl_path_rates.py (Parser p2p_readiness NUR Effektivwerte, sgl-project#278-TSV, NCCL-Referenz-Schema v1 mit Pflicht-p99+Last-Arm); harte Regeln je an einer Stelle erzwungen: Placeholder-Neutralitaet (ohne measured-Daten byte-identische sgl-project#240-Wahl, per Identitaets-Test ueber 3x3x4x4 belegt), Graph-Safety, Prio-Schutz, Offload-Latenz-Term-Anschluss; Bus-Begriff mit sgl-project#286-Arbiter geteilt via benanntem Sensor-Adapter; SGLANG_HTCCL_PATH_DISPATCHER Default aus; 46 neue Tests, distributed-Suite ohne neuen Fehlschlag
efschu added a commit to efschu/htsglang that referenced this pull request Jul 29, 2026
…terie (s00 preflight -> s01 P2P-Re-Probe -> s02-s05 R7c-Boots mit Report-Gate nach A -> s06 NCCL-Referenz -> s07 Register-GPU -> s08 Dispatcher-Tabellen+Neutralitaet -> s09 Sensor-Smoke; ~3h28 erwartet, harte Budgets je Schritt), run_step.sh mit Resume-Gate/Korridor/Locks/Timeout/py-spy/Verdikt, battery_state.py (state.json, Artefakt- statt Reihenfolge-Abhaengigkeiten, BLOCKED statt stillem Skip, --force/--only/--from mit history), 10 inhaltliche Checks (Positionskurve+Referenzspalte, echte sgl-project#279-Lader, CudaDeviceOps-Identitaet + parked-Zustandsfolge, werfende Hooks als Neutralitaetsbeweis), EXECUTOR_PROTOCOL (haiku Skript-/sonnet Boot-Schritte, FAIL/STOP = anhalten ohne Debug) + HANDOFF_TEMPLATE fuer den opus-Fixer; 121 hermetische Trockentests (Checks als Subprozess gegen PASS/FAIL/STOP-Fixtures, no-P2P-Rig-Toleranz gefixt); Vorbehalt dokumentiert: s06/s07/s09 sind neue Messskripte, Erstlauf beaufsichtigen
efschu added a commit to efschu/htsglang that referenced this pull request Jul 29, 2026
… integriert + all_gather-Stopper geschlossen — Merge-Ordnung in _select ist die Substanz (handles() sgl-project#240 -> refine sgl-project#279 -> lauter Riegel zuletzt, sonst entkaeme HINT_GLOO-in-Capture); all_gather als AG-Phase des a2a-Kerns (kein neuer Kernel: konstanter Sende-Offset, erbt Byte-Beleg/Runden-Paritaet/ragged-tail; Shard > Slot laeuft in ceil(shard/slot) Runden statt Ablehnung — 10.600.448 B = 2 Runden; graph-safe weil Rundenzahl gruppen-uniform eingebrannt, Direktmodus bewusst NICHT), uneven-Shards als Laengenvektor; reduce_scatter/broadcast bleiben hinterm Riegel mit begruendeter Meldung aus HTCCL_OPS statt Literalliste; drei Fork-Bugfixes als erste Commits (--help %-Maskierung, scheduler hicache, test_utils CVD=leer) mit Falsifikatoren; Bonus belegt offline: bar1_netz_kernel STACK 64->0 (ring/a2a byte-identisch); F811-Fund + Signatur-Fix; distributed 949 passed / 16 vorbestehend, Dispatcher-46 unveraendert; GPU-Checkliste in INTEGRATION_R3_VALIDATION §BAR1 + Runbook §4.15
efschu added a commit to efschu/htsglang that referenced this pull request Aug 2, 2026
Desk design only, no cards (CUDA_VISIBLE_DEVICES=99), no implementation.

Verdict: sgl-project#407 is not a new abstraction, it is a NODE layer plus a
reconciliation. The edge half — what does it cost to move a byte from here
to there — already exists FOUR times with four readers and four absent-value
rules: cost_model.PairMatrix (#348b, UUID-keyed), barlink_path_rates.
PathProfile (sgl-project#279, affine per route), offload_movement.PeerPathCapability
(sgl-project#286, CUDA-ordinal-keyed) and card_probe h2d/d2h (sgl-project#271). E2 and E3 both want
the effective BAR1 aperture from p2p_readiness/capability_matrix.json and only
E2 has a loader. What exists nowhere is the node layer, and in its place the
tree carries four disjoint tier vocabularies (PARK_TARGETS sgl-project#286;
LOCAL_DESTINATION/SUPPORTED_PARK_BACKENDS sgl-project#224; StorageMedium telemetry;
HiCache choices=). Expert offload, the largest byte mover in the fork, is in
none of them: its tier is device="cpu" + .pin_memory() written out five times.

Two shipped contradictions the design bends around, both stated up front:

C1 — two opposite total orders over one axis. sgl-project#224's destinations_error makes
'local' first a hard validation law ("a remote tier is a tier BELOW local,
never above it"); sgl-project#286's ladder puts peer_vram ABOVE host_ram. Resolved by
making the order a shortest path over a STAGING GRAPH: 'local first' is
recovered as reachability (the only edge out of a device tier toward a blob
tier runs through host), peer_vram is recovered because vram->vram is a direct
edge. Neither consumer's shipped behaviour changes.

C2 — the one existing tier ledger is positionally keyed. CapacityLedger keys
peer-VRAM budgets by CUDA ordinal; AUDIT_331 calls that SCOPED and legitimate
until it crosses a process boundary, which is exactly what sgl-project#305 §6 ("one
ledger, no second accounting") asks for. On this rig the 5090 is CUDA ordinal
0 and NVML index 1, so the failure is silent card-swapping.

Honesty correction on three numbers the brief assumed measured: the peer-VRAM
1-3 us posted-write class is an explicit ASSUMPTION (EVAL_p2p §"Tragende
Annahmen"), NVMe latency does not exist in the tree in any unit, and host DRAM
38 GB/s is derived from an ASSUMED DDR4-3200 peak (the VM hides SPD). They
ship as ESTIMATE/ABSENT under #348b's measured-or-named rule. Everything else
in the inventory is sourced: 5090 32607 MiB / 3080 20480 MiB, membw 1558/723
GB/s, BAR1 256 MiB nominal but 96 MiB contiguous with 1.13-1.34x vs NCCL at
20 KiB-16 MiB, H2D 6.4/13/13 GB/s gen4 x4/x8/x8, NVMe 1.8 GB/s cold (ARC rows
excluded), rig-2 2.07-2.83 GB/s with 1.47 us at 8 B and ~89/202 us serving
collectives.

Consumer survey, eight consumers read in their target-selection code:
sgl-project#77/sgl-project#123 expert offload MECHANICAL for the UUID-vector half (it revives sgl-project#394's
merged-but-starved link-proportional sharding, 145 -> 86 ms/token, 1.69x) and
REBUILD for a third rung (two dicts, binary resident|spill, UVA aliasing);
HiCache MECHANICAL for backends, ladder EXCLUDED (it is the type signature of
every controller method, not configuration); sgl-project#224 MECHANICAL for the list,
REBUILD for the ordering law, victim half untouched; sgl-project#89 MECHANICAL-PLUS
(needs a persistence class or a tmpfs --hibernate-dir is a silent regression,
and a driver-free arg-parse-time query); sgl-project#305 NO MIGRATION, it is the awaited
mechanism; sgl-project#286 MECHANICAL and the donor — lift _select_target and
CapacityLedger rather than write new ones; sgl-project#389 and sgl-project#306 greenfield.

Cut plan, no big bang: 1 registry + read-only enumeration + GateRow dashboard
row (S); 2 publish the rank->card-UUID vector without a collective so sgl-project#394
comes alive (S, the only cut whose yield is already measured); 3 sgl-project#224 consumes
it (S-M); 4 CapacityLedger becomes a view of registry/ledger.py, keys widen to
TierId (M); 5-9 expert offload, hibernate, sgl-project#305 transitions, HiCache backends,
NVMe/sgl-project#306 declaration. Exclusions with hard reasons only: the HiCache ladder,
GDN state (DEVICE_BOUND_ONLY admission, a refusal not an omission), cross-rig
GPU-to-GPU (driver guard, sgl-project#280 closed in userspace), compute placement (sgl-project#302).

Measurement plan names six gaps, five of them S, three of them new arms in the
existing comm-suite which today has no BAR1, storage or host-DRAM arm.

codespell clean; ruff not applicable (documentation only).
efschu added a commit to efschu/htsglang that referenced this pull request Aug 2, 2026
…r plus a reconciliation, not a new abstraction — edge costs already exist FOUR times (PairMatrix #348b, PathProfile sgl-project#279, PeerPathCapability sgl-project#286 on CUDA ordinals, card_probe sgl-project#271) with four absent-value rules and no reconciler, while four disjoint tier vocabularies coexist and the largest byte mover (expert offload) is in none of them; honesty corrections shipped as ESTIMATE/ABSENT (peer-VRAM us-class is an assumption, NVMe latency unmeasured in any unit, DRAM derived from assumed DDR4-3200); contradictions resolved in design: local-first vs peer-vram-above-host becomes shortest-path reachability over a staging graph (neither consumer changes), the positionally-keyed CapacityLedger named as the silent-card-swap hazard at process boundaries; TenantState frozen (client-visible enum mixing tiers into policy) with orthogonal resident_tier; cut plan S-first with cut 2 = publish rank->UUID vector WITHOUT a collective — revives sgl-project#394, the only cut with measured yield (1.69x) (sgl-project#407)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant