Skip to content

Add Together and AzureOpenAI examples - #184

Merged
merrymercy merged 2 commits into
mainfrom
together
Feb 12, 2024
Merged

merrymercy merged 2 commits into
mainfrom
together

Conversation

@merrymercy

Copy link
Copy Markdown
Collaborator

No description provided.

@merrymercy
merrymercy merged commit bb824da into main Feb 12, 2024
@merrymercy
merrymercy deleted the together branch February 12, 2024 09:06
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
rucnyz added a commit to rucnyz/sglang that referenced this pull request Jun 2, 2026
…tasks + T35 verify

Audit verdict per T (the 16 pending PLAN §3/§4 items):

DONE (covered by existing impl + tests):
  * T29 SGLANG_KV_POLICY_MODULE → _load_eviction_scorer exists;
    verify/t38/ stage C1 pins resolver
  * T32 all-rank atomic actions → tokenizer-server fanout exists;
    verify/t15/run_dp2_real.py exercises multi-rank wire
  * T35 authoritative_tier → baselines/base.py + new verify/t35/
    (8 stages: all residence combinations + empty-raises)

STRUCTURAL (compile-time invariant, no dedicated verify needed):
  * T19 atomic unit visibility → page_aligned() called in insert
    path at unified_radix_cache.py:574/599/731; run-time pin
    deferred to integration_stress flavor B

PARTIAL (impl half done):
  * T24 HASH_COLLISION → detection live with pair-dedupe; webhook
    firing pending ("T24 webhook pending" log) → sgl-project#182
  * T25 idempotent endpoints → migrate ✓, threshold ✓, pause/hint
    gated on T21/T40

OPEN — new tracking tasks created:
  * sgl-project#180  T18 state-dump single-snapshot read-lock
  * sgl-project#181  T21 PUT /aginfer/program_paused endpoint
  * sgl-project#182  T24 wire HASH_COLLISION webhook firing
  * sgl-project#183  T30+T39 proxy disconnect (paired)
  * sgl-project#184  T40 F2 hint emitter (daemon → sglang)
  * sgl-project#185  T41 F5 SESSION_END-for-PAUSED handler

OPEN — already tracked elsewhere:
  * T26 (HiCache/Mooncake EMA) → sgl-project#172 (T13 deferred)
  * T27 hint clear ordering → blocks on T40 (sgl-project#184)
  * T28 should_write_through → sgl-project#178

OUT OF SCOPE:
  * T31 harbor /aginfer/session_end — lives in harbor/client,
    not in sglang/daemon codebase

PLAN.md updated with per-T status notes referencing the audit
date + task IDs.  verify/t35/ added (8 stages, all green).
yyj6666667 added a commit to yyj6666667/sglang that referenced this pull request Jun 7, 2026
…t in MoE

The fast _varlen_deep_gemm_silu_mul_quant kernel computes plain
silu(gate)*up, but M3 trains with swiglu_oai:
    out = gate * sigmoid(gate * swiglu_alpha) * (up + 1)
with gate clamped at +swiglu_limit and up clamped at +/-swiglu_limit.
For M3 the config carries swiglu_alpha=1.702, swiglu_limit=7.0; the
model code already forwards these to FusedMoE as gemm1_alpha /
gemm1_clamp_limit and they live on MoeRunnerConfig. But the kernel
signature drops them, so the runner was silently running silu and
emitting un-clamped intermediate values; by the second MoE layer
gateup_output overflowed fp8 quantization, the down_input_scale
exploded to ~1e31, down_input became NaN, and from there the
hidden-state stream collapsed into argmax-of-NaN territory
(observed as deterministic 32 NUL tokens at temp=0 and incoherent
"p"-clustered tokens at temp>0).

Localized via hidden-state dump on debug branch:
  sgl-project#181 gateup            bf16 abs_mean=0.013 OK
  sgl-project#183 act.down_input    fp8  nan=True       FIRST NAN
  sgl-project#184 act.down_input_scale fp32 max=7.78e31 EXPLOSION

Fix without touching the Triton kernel: when MoeRunnerConfig has
gemm1_alpha set (i.e. M3-style), do the activation + per-token-group
fp8 quantization in eager PyTorch using the correct swiglu_oai math,
then feed the result into the existing fp8 grouped GEMM path. Slower
than a fused kernel but numerically correct; the fast path is kept
for non-M3 models.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
rucnyz added a commit to rucnyz/sglang that referenced this pull request Jun 7, 2026
…ints)

Per event the daemon re-scores the units in D_t and pushes ALL of
them to sglang via a fire-and-forget PUT /aginfer/hints, keeping the
inline scorer's V_u inputs warm without an HTTP round-trip at the
allocation callsite.  Full round-trip: daemon emitter + sglang
receiving side + overwrite-by-stamp storage.

Daemon:
* kv_scheduler.hints_from_state — one {hash, p_hat, lambda, stamp}
  per D_t unit, carrying the exact p_hat/lambda the scorer computed.
  stamp = int(sched_state.t) = sglang's own time_counter (monotonic,
  daemon-restart-surviving, read from the snapshot — NO wall-clock in
  the policy path).
* handle() dispatches hints via _dispatch_hints → enqueue_hints
  BEFORE and independent of the migrate decision, every event with a
  non-empty D_t.  NO shadow {hash: last_pushed} cache (DESIGN §10) —
  redundant re-pushes are absorbed by sglang's overwrite-by-stamp.

sglang (full chain, mirrors program_paused):
* PUT /aginfer/hints + _validate_hints_body (p_hat∈[0,1], lambda≥0,
  stamp non-negative int) → UpdateAginferHintsReq → rank fan-out →
  scheduler handler → UnifiedRadixCache.set_aginfer_hints.
* set_aginfer_hints overwrite-by-stamp: newer stamp wins; equal
  stamp = idempotent no-op (§10 R2); older = stale drop.  Plus
  get_aginfer_hint / clear_aginfer_hint; /aginfer/state echoes
  n_aginfer_hints (count, both dump paths).

Scope boundary (deferred, separate tasks): inline scorer CONSUMING
the table for eviction order, unit-birth seeding, eviction-time
clear ordering (T27), V_u-aware should_write_through (T28 sgl-project#178),
cross-rank fan-out atomicity (sgl-project#174).  This is emitter + storage +
overwrite contract only.

verify/t40: 14 stages incl. F0 LIVE e2e against a real sglang launch
(applied counts + cross-run idempotency + n_aginfer_hints readback).
Regression: kv_scheduler_value_rule / T36 / T41 / T6 /
integration_stress (full stack, 6 flavors) all green.
rucnyz added a commit to rucnyz/sglang that referenced this pull request Jun 7, 2026
… gaps

Adversarial audit found NO correctness bug in the shipped emitter /
overwrite-by-stamp logic; closed one validator hardening + five
test-depth findings (TDD: the C1 non-finite cases were red against
the pre-hardening validator, then green after).

* A4 — _validate_hints_body now rejects non-finite p_hat / lambda
  (math.isfinite).  The validator is the inline scorer's safety
  boundary; a NaN p_hat / inf lambda would corrupt eviction-order
  comparisons.  Defense-in-depth — the daemon never emits non-finite
  (p_hat∈[0,1], lambda=max(1e-3,·) or clamped), so not a live bug.
  Covered by t40 C1 (nan/inf p_hat + nan/inf lambda).
* #1 → G0 — dump-path parity guard: both _dump_aginfer_state_dict
  and _dump_aginfer_state_bytes emit n_aginfer_hints from the same
  self._aginfer_hints (the sgl-project#181 dump-divergence bug class).  NB the
  audit's "dict-only" claim was backwards — get_aginfer_state uses
  the BYTES path, which F0 already exercises end-to-end.
* #2 → H0 — outbound worker _post_one routes endpoint=hints via PUT
  (the one dispatch branch t36/t41 didn't cover for hints).
* A5 → D4 — mixed-batch applied accounting (newer+equal+new → 2),
  the realistic D_t re-push case.
* A11 → D5 — clear_aginfer_hint present/absent.
* B0 literal pin (p_hat == 1.0) kills the same-source tautology;
  A7 → F0 applied >= 2 (TP>1-safe; HTTP sums applied across ranks).

A6 (unbounded _aginfer_hints until eviction-clear) is an
acknowledged deferral — clear_aginfer_hint is the primitive, the
eviction-callsite ordering is T27 (tracking task sgl-project#188).

verify/t40: 18 stages (14 + 4), all green incl. F0 live e2e
(post-audit log: results/20260602_t40_post_audit_pass.log).
rucnyz added a commit to rucnyz/sglang that referenced this pull request Jun 7, 2026
…h-seed + clear)

The consumer side of the hint table (producer = sgl-project#184) — the heart of
"aginfer full" mode, where daemon-pushed p_hat/lambda actually drive
sglang's eviction order.  Three coupled parts:

1. Hint-aware scorer.  SGLANG_KV_POLICY_MODULE=aginfer:hint_v_u (a
   sentinel) binds UnifiedRadixCache._aginfer_eviction_score, a cache-
   bound method that reads _aginfer_hints by the node's own hash and
   computes the §7 V_u via the adapter's hint_v_u.  Architecture:
   bound method reads the dict (no hash->node index, single source of
   truth) but DELEGATES V_u to the adapter — hint_v_u shares
   _v_u_from_unit with ours_greedy_score, so one formula / no drift
   (verify/t27 A2 drift guard).  Absent hint -> local fallback, never
   bare LRU (DESIGN §3).
2. Unit-birth seeding.  _aginfer_seed_birth (p_hat~=1, never clobbers a
   daemon hint) from BOTH _add_new_node (leaf) AND _split_node (the new
   internal/shared-prefix node).  The live e2e exposed that without
   seeding split nodes ~10% of live units were uncovered; with it
   n_aginfer_hints == live_units (full DESIGN §3 coverage).
3. Eviction-clear ordering.  clear_aginfer_hint from
   _remove_leaf_from_parent — the single death/commit chokepoint
   (device-evict-death / host-evict / tombstone / migrate-DROP) — AFTER
   the detach, so DESIGN §10 ordering holds (scorer-read -> evict-commit
   -> clear).  This GC bounds _aginfer_hints to the live-unit set.

Atomicity (DESIGN §10 seqlock): satisfied trivially — sglang's single-
threaded scheduler serialises the /aginfer/hints PUT handler and the
eviction path, so the dict needs no lock; no CAS implemented (no
concurrency to guard).  DESIGN-vs-code note in verify/t27.

Default/non-aginfer mode is byte-identical: _init_aginfer_eviction_
scoring falls through to _load_eviction_scorer for non-sentinel specs;
the clear is guarded on a non-empty table; birth-seed is a no-op.

verify/t27: 13 stages + live GPU e2e (kv_policy_loaded=aginfer:hint_v_u,
no crash under serving; full coverage gap=0; 300 churned prefixes ->
table BOUNDED, no leak; hint PUT round-trip).  Regression: T28 / T38 /
T40 / integration_stress (6 flavors) green.

Deferred: the V_u-aware should_write_through (the sgl-project#178 hook's aginfer
registration) — sibling, same hint table.
efschu added a commit to efschu/htsglang that referenced this pull request Jul 26, 2026
…er multi-layer draft extend

STRUCTURAL FIX, GPU EVIDENCE STILL OPEN -- see the last paragraph.

MultiLayerEagleDraftWorker._draft_extend_for_decode runs one model runner per
MTP rung, each with its own attention backend
(init_attention_backend binds draft_runner_list[step].attn_backend =
draft_extend_attn_backend_list[step]). prepare_for_draft_extend plans the
metadata of exactly ONE of them -- the runner it is passed,
draft_runner_list[0] -- and the eager branch then calls
forward_batch.mark_forward_metadata_ready(), which makes every forward skip
its own init (forward_metadata_replan_equivalent stays False, so the staleness
re-plan does not kick in either). Rungs >= 1 therefore ran on whatever their
backend held from the last prefill draft-extend: metadata for a different
batch shape, or none at all. This is the defect behind the standing
"can't use cuda graph for draft extend! may have correctness issue!" warning.

The composite graph path does it right and is the template: its per-step
runner re-plans draft_extend_attn_backend_list[step] inside replay(), per rung,
right before that rung's graph runs. The eager loop now does the same, and for
the same reason keeps it per rung immediately before the forward rather than
planning all rungs up front (backends may share a planning workspace).

Guards preserve the two documented exceptions: NPU with --disable-cuda-graph
deliberately leaves each runner to init in forward_extend (post-pad), and idle
batches plan nothing, exactly as prepare_for_draft_extend does.

Tests (CPU, no GPU used):
  - test_multi_layer_eagle_draft_extend_decode.py: drives the real eager
    _draft_extend_for_decode with fake rungs that record every plan and
    forward. Pre-fix trace: plan0, forward0, forward1, forward2. Post-fix:
    plan0, forward0, plan1, forward1, plan2, forward2. Two ratchets pin the
    template (graph replay() still plans its rung) and the fix shape (the plan
    call indexes draft_extend_attn_backend_list).
  - test/registered/unit/spec/ + test_boot_constructor_integrity.py:
    610 passed, 1 failed. The failure
    (test_decode_bookkeeping_ownership::test_bookkeeping_sites_match_owner_allowlist,
    kv_session_offload.try_spill) is pre-existing: it fails identically on the
    untouched base commit.

NOT PROVEN: the numerical correctness of the eager multi-step path after this
fix. That needs a GPU and a multi-layer EAGLE checkpoint (MiMo-V2 or Step-3.5,
neither in the local model cache). The warning therefore STAYS in the code,
extended with a pointer to this fix, and is only to be removed once an A/B
against the graph path on real hardware shows matching accept lengths.
efschu added a commit to efschu/htsglang that referenced this pull request Jul 27, 2026
…ive-len + pre-existing bugs into integration/r3-probe

feat/mleagle-adaptive-len is an ANCESTOR of fix/mleagle-preexisting-bugs, so
this single merge carries the ordered pair the plan asked for:
  a8af70e sgl-project#138 adaptive draft length for multi-layer EAGLE
  d29a13e sgl-project#185 rank-0-broadcast the multi-layer EAGLE draft picks (sgl-project#50 family)
  3fb8884 sgl-project#184 plan per-rung attention metadata in the eager draft extend

Semantic pre-check on the doubly-touched server_args.py: regions are disjoint.
The tip's hunks sit at 1562 (sgl-project#202 --rank-tp-ratio x --pp-size guard) and 5691
(sgl-project#197 GGUF dense vocab); this branch's sit at 9552-9583, introducing the
adaptive_max_candidate_steps cached_property and routing the existing
draft-buffer sizing through it. No shared symbol, no shared validation path.

No file overlap at all with merge 4 (fix/spec-upstream-followups): that branch
is eagle_utils.py / multi_layer_eagle_draft_extend_cuda_graph_runner.py, this
one is multi_layer_eagle_utils.py / multi_layer_eagle_worker_v2.py /
eagle_worker_v2.py / spec_utils.py.
Chosen-David pushed a commit to Chosen-David/sglang that referenced this pull request Oct 9, 2026
…复完成)

- E117 addendum 补记:mavg 同方法口径 gap 中位 34.79% 判 GO,v1 NO-GO 降级为
  avg/avg 专属;回放 harness 抓到单池候选 α/β/γ 残留污染分数源缺陷(已修)
- PathMatrix addendum 补记:sgl-project#184 验收表落袋,三断言全属实,HF full 子格
  历史异构变体 + SG 投影 on unsupported 两个新边界
- Cascade addendum 补记:sgl-project#184 前置验收闭合,E120 可进入派发

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant