Skip to content

add awq marlin - #818

Closed
haichuan1221 wants to merge 2 commits into
sgl-project:mainfrom
haichuan1221:main
Closed

haichuan1221 wants to merge 2 commits into
sgl-project:mainfrom
haichuan1221:main

Conversation

@haichuan1221

Copy link
Copy Markdown

As the name suggests, add awq marlin from vllm

@zhyncs zhyncs left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May you benchmark the performance of BF16, AWQ, AWQ Marlin for Llama 3.1 8B Instruct? Thanks.

@zhyncs

zhyncs commented Jul 30, 2024

Copy link
Copy Markdown
Contributor

Maybe python3 -m sglang.bench_serving --backend sglang is ok.

@haichuan1221

Copy link
Copy Markdown
Author

May you benchmark the performance of BF16, AWQ, AWQ Marlin for Llama 3.1 8B Instruct? Thanks.

Yes, I will do so

@Ying1123

Copy link
Copy Markdown
Contributor

It seems you are updating some code that will not be used. What is wrong if you just load awq-marlin checkpoints?
#826 should be enough.

BTW, once you get performance numbers, please share them here.

@Ying1123 Ying1123 closed this Jul 30, 2024
@haichuan1221

Copy link
Copy Markdown
Author

ce you get performance numbers, please share t

Yes, you are right. Vllm use awq_marlin if awq can converted to marlin as shown: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/quantization/awq_marlin.py#L71.

efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 23, 2026
[sgl-project#818] fails fast in the BAR1 abort gate when a peer process is gone. Trivial:
this head carries c5a610b (the fix/814 merge) as an ancestor, already in
this branch since stage 1, so the merge contributes only barlink_bar1.py and
its new suite.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 23, 2026
…t#816, sgl-project#810, sgl-project#806, sgl-project#797, sgl-project#790/sgl-project#777, sgl-project#817/sgl-project#820, sgl-project#818

Merge-checklist duty for the nine stages on this branch. Entries follow each
section's own house style, and the mechanism text is taken from the commits'
measured evidence rather than restated from the ticket titles.

§3 KV backing relief + the allocator cap -- UPDATED IN PLACE rather than given
a second bullet, because sgl-project#814 and sgl-project#816 are follow-on defects OF the KvRowCap
mechanism that bullet already describes: the census reading the withheld block
as a leak (340262 of 465190 ids), the lift being reachable only from a cutover
(one boot at 26.8% of its id space for the life of the process), and exposure
exceeding the backing (417850 rows over 105413 committed, the device-side
assert in masked_set_kv_buffer_kernel).

§3 HiCache staging write-through ring (sgl-project#810), new bullet, plus its two
companion refusals -- the unbounded-file-tier refusal and the boot preflight
ledger entry, the latter being why 22.01 GB of MHATokenToKVPoolHost across
three PP ranks previously reached the preflight as nothing.

§7 BAR1 deadline + loud abort -- appended the sgl-project#818 peer-liveness half to the
existing narrative: the gate could wait forever on a peer that no longer
exists, and neither Bar1CollectiveStalled (reset by every resolved read) nor
defer_stall_for_building_peer (900 s off a build marker) caught it.

§12 Robustness canon -- three new families: contradictory-flag (sgl-project#806),
read-back-after-construction (sgl-project#797), denylist-of-reasons (sgl-project#817, sgl-project#820).

§18.3 hicache staging sizing (sgl-project#810) -- §18's own rule is that a merge adding a
reusable module adds its entry in the SAME merge, and this module had none.
Records the removal of fits_pinned_host_budget so it is not reintroduced.

§18.6 mamba carry instrument (sgl-project#767, gated by sgl-project#790) and flip break-even N
(sgl-project#777).

No existing entry was contradicted. Checked before writing: none of sgl-project#814,
sgl-project#810, sgl-project#806, sgl-project#772, sgl-project#797, sgl-project#790, sgl-project#777, sgl-project#817, sgl-project#820, sgl-project#818 had a catalog entry, and
the one sgl-project#677 line (§19.2, RESTORE-NEVER-REBUILD) describes a different
mechanism than the sgl-project#677 layout hold, so it is not stale and was left alone.

Gates for the tree this documents (hermetic, CUDA_VISIBLE_DEVICES=""):

  battery test/registered/unit/{managers,planner,server_args,mem_cache}
    baseline integ @ 78d27da       44 failed
    c56d238 (through sgl-project#818)         30 failed, 8452 passed   0 new ids, 14 fixed
    1c4eadb (through sgl-project#816)         30 failed, 8461 passed   0 new ids, same set
  test_barlink_abort_gate_liveness_818.py (outside the battery dirs)  10 passed
  ruff --select=F401,F821,UP037 and codespell: 0 new findings vs the same
    file set on 78d27da (16 ruff / 6 codespell exist identically on base)
  docs-only change; codespell clean on the catalog itself
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants