FastAPI-based working frontend by zhuohan123 · Pull Request #10 · vllm-project/vllm

zhuohan123 · 2023-03-27T06:19:00Z

Add a FastAPI-based frontend to cacheflow while keeping the old script working.

Remaining TODOs:

Add a README for the FastAPI frontend.
Rename the old script.
Add a gradio demo web frontend.

WoosukKwon

LGTM! Thanks for your effort.

* Add underlying functions * tests done

…sthrough Passthrough trust_remote_code

Wenxh/fp8 on a100 v1 pr

…x faster flush The sinkhorn program keeps the whole [R, C] fp32 tile plus a working copy live; at 4 warps that is several KB of registers per thread, so the compiler spills to CUDA local memory and the driver permanently reserves local_bytes x max_threads x num_SMs of device memory for the context (~2 GiB for the [256, 128] head_dim-256 tile on a 188-SM part). This was silently charged to vLLM's memory profile and shrank the derived KV-cache capacity (issue vllm-project#10). 8 warps: ~70% less reserved local memory AND ~4x faster flush (2.50 -> 0.61 ms for a 256-tile batch — the spills were also the kernel's bottleneck; 16 warps saves a bit more memory but is 2x slower than 8). Balanced-tile output unchanged within fp32 reduction noise (~5e-7 rel); Qwen3-4B kvarn-vs-fp16 greedy decode remains 100% token-identical.

Without an explicit --block-size, the generic get_preferred_block_size fallback returns the MINIMUM supported kernel block size (64) whenever the framework default (16) is unsupported. A g128 preset then builds its cache with 64-token kernel blocks and dies on the 'KVarN requires block_size (64) == group (128)' assert — e.g. any hybrid-model deployment without spec decode (with MTP the hybrid block-size reconciliation happened to land on 128, which is why issue vllm-project#10 reporters never hit it; explicit --block-size 128 also avoids it). Override get_preferred_block_size on the KVarN backend to return the active preset's group. Validated: Qwen3.6-27B TP2 without spec decode now starts ('Setting kv cache block size to 128 for KVARN backend') and serves; with-MTP behaviour unchanged.

…ffers Follow-up to the issue vllm-project#10 capacity fixes — recovers another ~1.6 GiB of KV budget by making the startup memory accounting honest: 1. Materialize ALL of KVarN's lazy state (per-layer fp16 tail pools, shared decode scratch, kernel JIT + autotune) inside the worker's memory-profiling window. The dummy profile run skips attention (attn_metadata=None), so this state previously first ran inside the CUDA-graph memory estimation warmup and was charged to the 'graph memory' estimate (1.7 GiB mislabeled; the real graph pool is ~54 MiB), with the pools additionally double-counted against the explicit worker reservation. The arithmetic pool reservation is dropped — the allocation is now real and measured. Accounting stays correct with VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. 2. Size the split-K partial buffers (mid_o / mid_lse) by what the split-K decode path can actually index — N = max_num_seqs x Hq rows — instead of the max_num_batched_tokens-driven q_rows (~85x oversized: 256 MiB -> ~3 MiB at typical configs). The decode driver additionally falls back to the single-stage kernel if a (padded dummy) batch ever exceeds the buffer rows. Qwen3.6-27B AWQ TP2 (issue vllm-project#10 configs), per rank: - CUDA graph estimate 1.76 -> 0.08 GiB (= fp8's), available KV 5.2 -> 6.77 GiB, capacity 540,503 -> 699,951 tokens (1.87x fp8's 373,445; the original report was ~0.1x). - without spec decode: 774,516 tokens, decode 70-72 tok/s coherent (exercises the resized split-K path); with MTP: 49 tok/s coherent, 8-way concurrent 0 errors. - Qwen3-4B kvarn-vs-fp16 greedy regression: 100% token-identical.

…project#10 loops) Prefix caching shares/recycles blocks across requests; the dense builder kept per-request flush state and discarded finished requests' pool data, so cache hits read stale tiles and an empty first block — surfacing as repetition loops. Reworked to a per-block, sharing-safe lifecycle (same design as the MLA fix 2e8f5cad7): reclaim flushes complete blocks, sinks stay fp16-resident and are evicted lazily. Cache-hit replay now byte-identical on the reporter's Qwen3.6-27B-AWQ setup, with and without MTP; repro + stress scripts in scripts_kvarn_dense/.

…lm-project#10) Fixes the two regressions reported after the MTP fix (d7bbff123): decode collapsing to <5 tok/s, and KV-cache capacity ~10x below fp8. 1. _cached_multiquery_path: a spec-decode verify step (or chunked-prefill continuation) now builds the batch's rotated fp16 K/V with the ONE block_table-driven Triton kernel and runs a single flash_attn_varlen call. FA's varlen causal mask is bottom-right aligned for seqlen_q < seqlen_k — token t attends keys <= cached_len + t — exactly the verify semantics, so no explicit mask is needed. Replaces _decode_path_slow on this route (per-request Python gather with per-block .item() syncs + Python dequant + fp32 SDPA, per layer, per step). The slow path remains the fallback for head_dim > 256 or a batch whose total KV exceeds the materialize scratch. 2. _warm_decode_kernels: compile + autotune the fused decode kernel, split-K stage1/2, and the packed-KV build kernel at profile time on tiny synthetic state. They never ran during vLLM's prefill-shaped profiling, so their one-time JIT/autotune cost (and, before fix 1, the slow path's fp32 materializations) landed in the CUDA-graph memory estimation warmup and was charged as graph memory — GiBs off the derived KV capacity. Qwen3.6-27B AWQ TP2 + MTP(3) + prefix caching + chunked prefill (the issue vllm-project#10 configs): - KV capacity 92,800 -> 540,503 tokens (fp8 baseline: 373,445; KVarN is now 1.45x fp8 instead of 0.25x); --max-model-len -1 fits the full 262K context again. - decode 43-47 tok/s single stream, 205 tok/s at 8 concurrent, coherent output, 0 errors (was <5 tok/s). - Qwen3-4B no-spec regression: kvarn-vs-fp16 greedy decode 100% token-identical.

Under MTP every decode step is a verify step, and the previous route materialized the request's ENTIRE context into fp16 scratch (then ran flash attention over it) on every step, per layer — an O(context) per-step cost that dominated long-context decode (issue vllm-project#10 follow-up: 5-12 tok/s on long agent sessions). Both fused dual-source decode kernels (single-stage and split-K stage 1) gain a VQ_INDIRECT mode: one program per (verify token, KV head), with a block-table row indirection and a per-token bottom-right causal length, so the verify step reads int4 tiles and the fp16 tail pool directly — no materialization. New driver kvarn_verify_attention; the backend routes by context depth (KVARN_FUSED_VERIFY_MIN_BLOCKS, default 64 blocks): the materialize route stays for short contexts, where its single round-trip is cheap and measured faster, and for chunked-prefill continuations, where one materialization amortizes over thousands of query tokens. KVARN_FUSED_VERIFY=0 forces the previous behavior. The kernel warmup covers the new specializations so the first verify step does not JIT. Measured (Qwen3.6-27B AWQ, single GPU, single stream, greedy, MTP k=3): decode 45.2 -> 51.6 tok/s at 32K context (+14%), parity at shorter contexts; the advantage grows with depth since the kernel reads ~4x fewer bytes than the fp16 round-trip. MTP long generation stays coherent, and the non-speculative path is byte-identical (Qwen3-4B greedy battery) since existing launch sites pass VQ_INDIRECT=False.

…x faster flush The sinkhorn program keeps the whole [R, C] fp32 tile plus a working copy live; at 4 warps that is several KB of registers per thread, so the compiler spills to CUDA local memory and the driver permanently reserves local_bytes x max_threads x num_SMs of device memory for the context (~2 GiB for the [256, 128] head_dim-256 tile on a 188-SM part). This was silently charged to vLLM's memory profile and shrank the derived KV-cache capacity (issue vllm-project#10). 8 warps: ~70% less reserved local memory AND ~4x faster flush (2.50 -> 0.61 ms for a 256-tile batch — the spills were also the kernel's bottleneck; 16 warps saves a bit more memory but is 2x slower than 8). Balanced-tile output unchanged within fp32 reduction noise (~5e-7 rel); Qwen3-4B kvarn-vs-fp16 greedy decode remains 100% token-identical.

Without an explicit --block-size, the generic get_preferred_block_size fallback returns the MINIMUM supported kernel block size (64) whenever the framework default (16) is unsupported. A g128 preset then builds its cache with 64-token kernel blocks and dies on the 'KVarN requires block_size (64) == group (128)' assert — e.g. any hybrid-model deployment without spec decode (with MTP the hybrid block-size reconciliation happened to land on 128, which is why issue vllm-project#10 reporters never hit it; explicit --block-size 128 also avoids it). Override get_preferred_block_size on the KVarN backend to return the active preset's group. Validated: Qwen3.6-27B TP2 without spec decode now starts ('Setting kv cache block size to 128 for KVARN backend') and serves; with-MTP behaviour unchanged.

…ffers Follow-up to the issue vllm-project#10 capacity fixes — recovers another ~1.6 GiB of KV budget by making the startup memory accounting honest: 1. Materialize ALL of KVarN's lazy state (per-layer fp16 tail pools, shared decode scratch, kernel JIT + autotune) inside the worker's memory-profiling window. The dummy profile run skips attention (attn_metadata=None), so this state previously first ran inside the CUDA-graph memory estimation warmup and was charged to the 'graph memory' estimate (1.7 GiB mislabeled; the real graph pool is ~54 MiB), with the pools additionally double-counted against the explicit worker reservation. The arithmetic pool reservation is dropped — the allocation is now real and measured. Accounting stays correct with VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0. 2. Size the split-K partial buffers (mid_o / mid_lse) by what the split-K decode path can actually index — N = max_num_seqs x Hq rows — instead of the max_num_batched_tokens-driven q_rows (~85x oversized: 256 MiB -> ~3 MiB at typical configs). The decode driver additionally falls back to the single-stage kernel if a (padded dummy) batch ever exceeds the buffer rows. Qwen3.6-27B AWQ TP2 (issue vllm-project#10 configs), per rank: - CUDA graph estimate 1.76 -> 0.08 GiB (= fp8's), available KV 5.2 -> 6.77 GiB, capacity 540,503 -> 699,951 tokens (1.87x fp8's 373,445; the original report was ~0.1x). - without spec decode: 774,516 tokens, decode 70-72 tok/s coherent (exercises the resized split-K path); with MTP: 49 tok/s coherent, 8-way concurrent 0 errors. - Qwen3-4B kvarn-vs-fp16 greedy regression: 100% token-identical.

…project#10 loops) Prefix caching shares/recycles blocks across requests; the dense builder kept per-request flush state and discarded finished requests' pool data, so cache hits read stale tiles and an empty first block — surfacing as repetition loops. Reworked to a per-block, sharing-safe lifecycle (same design as the MLA fix 2e8f5cad7): reclaim flushes complete blocks, sinks stay fp16-resident and are evicted lazily. Cache-hit replay now byte-identical on the reporter's Qwen3.6-27B-AWQ setup, with and without MTP; repro + stress scripts in scripts_kvarn_dense/.

…lm-project#10) Fixes the two regressions reported after the MTP fix (d7bbff123): decode collapsing to <5 tok/s, and KV-cache capacity ~10x below fp8. 1. _cached_multiquery_path: a spec-decode verify step (or chunked-prefill continuation) now builds the batch's rotated fp16 K/V with the ONE block_table-driven Triton kernel and runs a single flash_attn_varlen call. FA's varlen causal mask is bottom-right aligned for seqlen_q < seqlen_k — token t attends keys <= cached_len + t — exactly the verify semantics, so no explicit mask is needed. Replaces _decode_path_slow on this route (per-request Python gather with per-block .item() syncs + Python dequant + fp32 SDPA, per layer, per step). The slow path remains the fallback for head_dim > 256 or a batch whose total KV exceeds the materialize scratch. 2. _warm_decode_kernels: compile + autotune the fused decode kernel, split-K stage1/2, and the packed-KV build kernel at profile time on tiny synthetic state. They never ran during vLLM's prefill-shaped profiling, so their one-time JIT/autotune cost (and, before fix 1, the slow path's fp32 materializations) landed in the CUDA-graph memory estimation warmup and was charged as graph memory — GiBs off the derived KV capacity. Qwen3.6-27B AWQ TP2 + MTP(3) + prefix caching + chunked prefill (the issue vllm-project#10 configs): - KV capacity 92,800 -> 540,503 tokens (fp8 baseline: 373,445; KVarN is now 1.45x fp8 instead of 0.25x); --max-model-len -1 fits the full 262K context again. - decode 43-47 tok/s single stream, 205 tok/s at 8 concurrent, coherent output, 0 errors (was <5 tok/s). - Qwen3-4B no-spec regression: kvarn-vs-fp16 greedy decode 100% token-identical.

Under MTP every decode step is a verify step, and the previous route materialized the request's ENTIRE context into fp16 scratch (then ran flash attention over it) on every step, per layer — an O(context) per-step cost that dominated long-context decode (issue vllm-project#10 follow-up: 5-12 tok/s on long agent sessions). Both fused dual-source decode kernels (single-stage and split-K stage 1) gain a VQ_INDIRECT mode: one program per (verify token, KV head), with a block-table row indirection and a per-token bottom-right causal length, so the verify step reads int4 tiles and the fp16 tail pool directly — no materialization. New driver kvarn_verify_attention; the backend routes by context depth (KVARN_FUSED_VERIFY_MIN_BLOCKS, default 64 blocks): the materialize route stays for short contexts, where its single round-trip is cheap and measured faster, and for chunked-prefill continuations, where one materialization amortizes over thousands of query tokens. KVARN_FUSED_VERIFY=0 forces the previous behavior. The kernel warmup covers the new specializations so the first verify step does not JIT. Measured (Qwen3.6-27B AWQ, single GPU, single stream, greedy, MTP k=3): decode 45.2 -> 51.6 tok/s at 32K context (+14%), parity at shorter contexts; the advantage grows with depth since the kernel reads ~4x fewer bytes than the fp16 round-trip. MTP long generation stays coherent, and the non-speculative path is byte-identical (Qwen3-4B greedy battery) since existing launch sites pass VQ_INDIRECT=False.

…ct#10) On sm120 the DeepGEMM paged MQA logits kernel only accepts block_kv 64 for the fp8 indexer cache, but the default alignment landed on 32 and aborted GLM-5.3-Flash during cuda graph capture. Align to the largest pool page on capability family 120 so the page split lands on 64. Signed-off-by: Aditya Jha <118803119+ima-helikoptaaa@users.noreply.github.com>

同步 ppu-ci-gha 脚本时覆盖了 336519a（2026-08-31 nightly 首跑修复），按原注释与格式补回： - Step 1 standard：MiniCPM4.1-8B 两个 use_prompt_embeds 变体（trust_remote_code 模型 NAS 缺动态模块代码，HF_HUB_OFFLINE=1 初始化即失败） - Step 2 extra_standard：bloom-560m 两个变体（疑似 PPU 数值漂移）、 Qwen2.5-1.5B-apeach classification、bge-base-en-v1.5 embedding （疑似 PPU Issue #10 同族 pooling 数值问题）已验证：check-exclusions.py 全量自检通过（167 条引用路径均存在） Co-authored-by: Qoder

@14k

…_dot4 (+27 % prefill) Opt-in with VLLM_RDNA_MOE_W4A8=1. The stock W4A16 Triton MoE kernel dequantises every int4 weight to fp16 inside the K loop and runs at ~4 TFLOPS on a V620 (~11 % of the measured fp16 dot rate; 28 % of prefill time). The new kernel quantises activations per token to int8, unpacks weights with integer ops, accumulates int8 x int8 -> int32 per 128-wide group (v_dot4_i32_i8) and folds the scales in once. Hooked at the top of invoke_fused_moe_wna16_triton_kernel so the modular TritonExperts path uses it; decode-shaped configs keep the stock path. Swept tiles 64x128x16 / 8 warps / 2 stages (BK >= 64 spills past 256 VGPRs): 2.0-2.45x on the MoE GEMMs, exact vs a torch reference of the same maths. In-server at 120 W: prefill 908 -> 1160 t/s @3.3k, 957 -> 1217 @14k, 962 -> 1232 @26K; decode unchanged; validate PASS; teacher-forced NLL within noise (-0.1 % / -0.2 %). CHANGES.md vllm-project#10. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…t, padded blocks) (vllm-project#10) Two bugs made MooncakeStoreConnector unusable on Kimi K3.1 (KDA + merged DSA cache, TP8): 1. Hash unit mismatch. The K3.1 platform hook sets prefix_match_unit=64, but it runs only in worker processes; the engine core never sees it and hashes requests per 1088-token block. The store worker re-resolved the hash unit from its own config (64), so every save tripped `token_len // hash_block_size <= len(block_hashes)` and nothing was stored. Use the core-resolved kv_cache_config.hash_block_size instead. 2. Padded kernel blocks registered as overlapping regions. The merged DSA cache pads each 64-token kernel block (64 x 788 B -> 50512 B). After grouping to 1088-token manager blocks, cache[0] is not dense, so the layer was split "per head" along the kernel-block dim: 17 regions, each a whole manager block long and offset by one kernel block. Every put moved 17x the data and every get overwrote the next blocks, i.e. other requests' KV. Decide block compactness on the kernel-block view before grouping, so a padded manager block moves as one region. Verified on 8xB300 with Kimi-K3.1 TP8 (96 greedy prompts, ~3.2k tokens): pass 1, reset GPU prefix cache, pass 2 loads from Mooncake. Before: pass 2 51/96 correct. After: 96/96 in both passes, identical outputs, 113k external prefix-cache hit tokens. Signed-off-by: zjy0516 <riverclouds.zhu@qq.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

zhuohan123 added 6 commits March 26, 2023 12:41

modify current frontend & server to prepare for real frontend

521dcfa

add server_utils

3e8c132

fix small bugs

8873415

add a new server class

71c2b93

initial implementation of fastapi frontend

d262ac9

fix memory bugs and add test client

ea06325

zhuohan123 requested a review from WoosukKwon March 27, 2023 06:19

zhuohan123 added 7 commits March 27, 2023 09:09

Rename and small fixes

03f09f0

Merge branch 'main' into real-frontend

16e9674

Rename and update readme

f045128

fix

6f735fa

fix api and add gradio webserver

f1666e8

Modify readme

9b2972f

Update README

9d23237

WoosukKwon approved these changes Mar 28, 2023

View reviewed changes

Comment thread cacheflow/sampling_params.py Resolved

Comment thread cacheflow/master/server.py Resolved

Address review comments.

739f599

zhuohan123 merged commit 721fa3d into main Mar 29, 2023

zhuohan123 deleted the real-frontend branch March 29, 2023 06:49

TheBloke mentioned this pull request Jul 20, 2023

Can't launch OpenAI API server on newly installed vLLM in Docker - fastchat not found #537

Closed

xiangyuT pushed a commit to xiangyuT/vllm that referenced this pull request Oct 25, 2023

Clean unused KVCache after usage (vllm-project#10)

fbcfee9

* Add underlying functions * tests done

shanshanpt mentioned this pull request Nov 17, 2023

Run long conetxt error : CUDA error: an illegal memory access was encountered #1700

Closed

junior-zsy mentioned this pull request Nov 20, 2023

Error with 32k Long Text in chatglm2-6b-32k Model #1725

Closed

hongxiayang pushed a commit to hongxiayang/vllm that referenced this pull request Feb 13, 2024

FastAPI-based working frontend (vllm-project#10)

e1f8fcc

slyalin pushed a commit to slyalin/vllm that referenced this pull request Mar 22, 2024

Merge pull request vllm-project#10 from slyalin/trust_remote_code_pas…

a398824

…sthrough Passthrough trust_remote_code

yuhuixu1993 mentioned this pull request Jun 2, 2024

[Bug]: loading squeezellm model #5190

Closed

ykim362 pushed a commit to ykim362/vllm that referenced this pull request Jun 17, 2024

Merge pull request vllm-project#10 from wenxcs/wenxh/fp8-on-a100-v1-pr

94d4614

Wenxh/fp8 on a100 v1 pr

alixiaodi mentioned this pull request Aug 2, 2024

[Bug]: #7072

Closed

morrison-turnansky mentioned this pull request Aug 27, 2026

[Bugfix] Remove Coincidentally-Correct Tests #54107

Open

4 tasks

Peuqui mentioned this pull request Aug 28, 2026

[Bug]: PP + async scheduling + spec decode: output_token_ids trim only runs on the last PP rank, desynchronizing the ranks until NCCL deadlocks #54260

Open

webdevtodayjason mentioned this pull request Aug 29, 2026

[Model] add GLM-5.3-Flash support #53906

Merged

4 tasks

alansrobotlab2 mentioned this pull request Sep 3, 2026

[Bug]: VLLM_PLE_CPU_OFFLOAD deadlocks at kernel warmup on single GPU (TP=1) - Qwen3.8-Flash-Next on GB10/sm_121 #53960

Open

askliar mentioned this pull request Sep 7, 2026

[Mamba] Unify FI ReplaySSM STP/MTP/Prefix Caching Lifecycle #55688

Merged

2 of 6 tasks

Ronald1995 mentioned this pull request Sep 22, 2026

[Frontend][RL] Track HTTP weight operation outcomes and concurrency #55781

Merged

Dazzanova mentioned this pull request Sep 25, 2026

[Bugfix] Join Anthropic system blocks with a separator instead of "" #58682

Open

3 of 4 tasks

xzwgit mentioned this pull request Sep 30, 2026

[Bugfix][Kernel] Bump DeepGEMM pin for SM120 32-state page support (DeepGEMM#14) #59385

Closed

lisawym mentioned this pull request Oct 1, 2026

[Bugfix][Kernel] Bump DeepGEMM pin for SM107 JIT compatibility #59503

Open

5 of 6 tasks

Lei-010 mentioned this pull request Oct 9, 2026

[Bug/Perf] DeepSeek-V4.1-Flash on SM120 (RTX PRO 6000 Blackwell, 8xTP): extremely low decode throughput with --enforce-eager; CUDA graphs unusable — please prioritize SM120 graph capture fix #56892

Open

ISEEKYAN mentioned this pull request Oct 21, 2026

[DSv4.1][BI] Bitwise batch invariance for DeepSeek-V4.1-Flash with CUDA graphs #62365

Open

tjtanaa mentioned this pull request Oct 26, 2026

[RFC]: Attention Backend Test Suites #63640

Open

1 task done

NickLucche mentioned this pull request Nov 6, 2026

[Mooncake] Lookup breaker follow-up: simplify config, don't count queue expiry, single probe hj-mistral/vllm#5

Merged

Navigation Menu

Repository navigation

FastAPI-based working frontend - #10

FastAPI-based working frontend#10
zhuohan123 merged 14 commits into
mainfrom
real-frontend

zhuohan123 commented Mar 27, 2023 •

edited

Loading

Uh oh!

WoosukKwon left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

zhuohan123 commented Mar 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

WoosukKwon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zhuohan123 commented Mar 27, 2023 •

edited

Loading