Skip to content

Add CoT feature for LLava in SGLang with demo - #365

Closed
fisher75 wants to merge 2 commits into
sgl-project:mainfrom
fisher75:main
Closed

fisher75 wants to merge 2 commits into
sgl-project:mainfrom
fisher75:main

Conversation

@fisher75

Copy link
Copy Markdown

I add a CoT(Chain of Thought) feature into SGLang with the use and inspiration of s+=.
In this case, the information of the picture and the dialog of the questions and answers can be used for the latest question.
The result demonstrated a clear CoT.

Refined and added demo
https://github.com/sgl-project/sglang/pull/348

@fisher75

fisher75 commented Apr 14, 2024 •

Copy link
Copy Markdown
Author

Demo:

questions_cot.jsonl:
{"image": "demo_cot.jpg", "text1": "Identify potential reasons why a city with yellow taxis would be experiencing a scenario where a person is ironing clothes on top of a cab.", "text2": "Considering the identified reasons, analyze how this situation could impact the traffic flow and city dynamics.", "category": "cot", "question_id": 0}

answers_cot.jsonl:
{"question_id": 0, "prompt1": "Identify potential reasons why a city with yellow taxis would be experiencing a scenario where a person is ironing clothes on top of a cab.", "text1": "There could be several reasons for this scenario in a city with yellow taxis. One possibility is that the person is a street vendor who uses the taxi as a makeshift workspace to iron clothes for customers. Another reason could be that the person is a taxi driver who is taking a break and using the taxi as a temporary ironing station. It could also be a situation where the person is in a hurry and has no other place to iron their clothes, so they are using the taxi as a convenient spot. Lastly, it could be a form of street performance or art, where the person is ironing clothes on top of the taxi to attract attention and create a unique experience for passersby.", "prompt2": "Considering the identified reasons, analyze how this situation could impact the traffic flow and city dynamics.", "text2": "The situation of a person ironing clothes on top of a yellow taxi could have several impacts on the traffic flow and city dynamics. \n\n1. Traffic flow: If the person is a street vendor, it could potentially slow down the traffic flow as they may need to move around the taxi to attend to customers. This could lead to congestion and delays for other vehicles on the road.\n\n2. Safety concerns: Ironing clothes on top of a moving taxi could pose safety risks for both the person and other road users. The person might lose balance and fall off the taxi, causing injury or even fatality. Additionally, the taxi driver may be distracted by the ironing activity, which could lead to accidents.\n\n3. Noise and pollution: The ironing process could generate noise and pollution, which could affect the overall ambiance of the city. The noise could disturb nearby residents and businesses, while the pollution generated by the ironing process could contribute to air and noise pollution in the area.\n\n4. Impact on city infrastructure: The presence of a person ironing clothes on top of a taxi could potentially damage the taxi's roof or other parts of the vehicle, leading to costly repairs for the taxi owner.\n\n5. Public perception: This unusual sight could create a negative impression of the city's public order and safety, potentially affecting tourism and the overall image of the city.\n\nIn conclusion, while the situation of a person ironing clothes on top of a yellow taxi might seem amusing or unique, it could have several negative impacts on traffic flow, safety, noise and pollution, city infrastructure, and public perception.", "model_id": "llava-v1.5-7b", "answer_id": 0, "metadata": {}}

@merrymercy

merrymercy commented May 12, 2024 •

Copy link
Copy Markdown
Collaborator

@fisher75 Thanks for the example. However, a more precise name for this example should be "multi-turn" instead of "CoT". Given this example does not add too many new values and is not well organized (e.g., Chinese comments, no readme, personal paths), I will close this for now.
Feel free to reopen if you have any updates.

@merrymercy merrymercy closed this May 12, 2024
shiyu7 pushed a commit to shiyu7/sglang that referenced this pull request Dec 26, 2025
efschu added a commit to efschu/htsglang that referenced this pull request Jul 31, 2026
…ess-suite reserve trap, text-identity-is-no-instrument tier note (sgl-project#365)
efschu added a commit to efschu/htsglang that referenced this pull request Aug 1, 2026
…the DEGRADED layout

The TP>num_kv_heads window (4 arms, ~20 min) confirmed Q1 to the digit and
inverted Q2, so the flag text now states both sides of the threshold with the
evidence for each.

Q1: the draft pool shards exactly at the predicted factor -- 655520 rows on
every rank (replicated) -> 307290/174131/174131 (dcp), K 0.31 -> 0.15/0.08/0.08
GB, total draft KV 1.86 -> 0.62 GB = -67%, which is exactly
1-(30+17+17)/(64*3). Identical on both vehicles, as expected: the draft pool
follows the vector, not the model size.

Q2, the discovery: ABOVE the threshold 'replicated' is not the cheaper setting,
it is the BROKEN one. At kv < tp the heads cannot be split, so a draft on the
replicated layout is asked for a head shard that does not exist -- the verified
output stays correct (the target governs it; output ids identical on every
graded pair) but its proposals are almost never accepted. Qwen3.5-2B: 61 verify
rounds for 64 tokens, accept 1.05, i.e. speculation paying full cost for
nothing, against 20 rounds / accept 3.20 on dcp. 35B-A3B the same shape, 32 ->
20. chain_quality_gate GREEN full marks on both vehicles.

So the help text no longer says 'dcp pays here'; it says which setting is
correct on each side of one threshold, and why:
  TP >  num_kv_heads -> dcp; replicated is DEGRADED (the 1.05-accept evidence)
  TP <= num_kv_heads -> replicated; dcp is VRAM-neutral and costs 10-16% accept

H2 (_replicated_kv_ragged_reindex) does not fire: the path is live per the
geometry log and short determined prompts generate correctly first-chunk.

PROBE-DESIGN RULE, folded into the doc and the runsheet because it generalises
beyond this task: a short probe used as a CORRUPTION check must have a
DETERMINED answer. The runsheet's original 'Hi.' probe was wrong -- at
temperature 0 an underdetermined prompt is rerouted entirely by a single
last-bit flip at position 0, so both arms stay fluent and the probe can only
ever report 'different', never 'wrong'. That is the instrument sgl-project#360/sgl-project#365 ruled
out, rebuilt by accident. Corrected form: mechanical sequences
(a b c d e f -> g h i j k l ...) or single-answer facts. Two further runsheet
corrections recorded: the sgl-project#328 gate may live on a newer commit than a feature
branch's base (run it from a tree that has it), and helpers are killed by
recorded pid, never by pkill -f.

distributed + spec + server_args: zero failures unique to this branch.
ruff/codespell/black clean. Help text and docs only.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants