Repository navigation
Conversation
|
Demo: questions_cot.jsonl: answers_cot.jsonl: |
|
@fisher75 Thanks for the example. However, a more precise name for this example should be "multi-turn" instead of "CoT". Given this example does not add too many new values and is not well organized (e.g., Chinese comments, no readme, personal paths), I will close this for now. |
…ess-suite reserve trap, text-identity-is-no-instrument tier note (sgl-project#365)
…the DEGRADED layout The TP>num_kv_heads window (4 arms, ~20 min) confirmed Q1 to the digit and inverted Q2, so the flag text now states both sides of the threshold with the evidence for each. Q1: the draft pool shards exactly at the predicted factor -- 655520 rows on every rank (replicated) -> 307290/174131/174131 (dcp), K 0.31 -> 0.15/0.08/0.08 GB, total draft KV 1.86 -> 0.62 GB = -67%, which is exactly 1-(30+17+17)/(64*3). Identical on both vehicles, as expected: the draft pool follows the vector, not the model size. Q2, the discovery: ABOVE the threshold 'replicated' is not the cheaper setting, it is the BROKEN one. At kv < tp the heads cannot be split, so a draft on the replicated layout is asked for a head shard that does not exist -- the verified output stays correct (the target governs it; output ids identical on every graded pair) but its proposals are almost never accepted. Qwen3.5-2B: 61 verify rounds for 64 tokens, accept 1.05, i.e. speculation paying full cost for nothing, against 20 rounds / accept 3.20 on dcp. 35B-A3B the same shape, 32 -> 20. chain_quality_gate GREEN full marks on both vehicles. So the help text no longer says 'dcp pays here'; it says which setting is correct on each side of one threshold, and why: TP > num_kv_heads -> dcp; replicated is DEGRADED (the 1.05-accept evidence) TP <= num_kv_heads -> replicated; dcp is VRAM-neutral and costs 10-16% accept H2 (_replicated_kv_ragged_reindex) does not fire: the path is live per the geometry log and short determined prompts generate correctly first-chunk. PROBE-DESIGN RULE, folded into the doc and the runsheet because it generalises beyond this task: a short probe used as a CORRUPTION check must have a DETERMINED answer. The runsheet's original 'Hi.' probe was wrong -- at temperature 0 an underdetermined prompt is rerouted entirely by a single last-bit flip at position 0, so both arms stay fluent and the probe can only ever report 'different', never 'wrong'. That is the instrument sgl-project#360/sgl-project#365 ruled out, rebuilt by accident. Corrected form: mechanical sequences (a b c d e f -> g h i j k l ...) or single-answer facts. Two further runsheet corrections recorded: the sgl-project#328 gate may live on a newer commit than a feature branch's base (run it from a tree that has it), and helpers are killed by recorded pid, never by pkill -f. distributed + spec + server_args: zero failures unique to this branch. ruff/codespell/black clean. Help text and docs only.
I add a CoT(Chain of Thought) feature into SGLang with the use and inspiration of
s+=.In this case, the information of the picture and the dialog of the questions and answers can be used for the latest question.
The result demonstrated a clear CoT.
Refined and added demo
https://github.com/sgl-project/sglang/pull/348