Fix hybrid_linear_attn_backend crash with ngram speculation

[he-yufeng]

Problem

hybrid_linear_attn_backend accesses spec_info.topk at runtime during target_verify mode, but NgramVerifyInput doesn't define topk, causing an AttributeError crash with --speculative-algo NGRAM.

Fix

Read topk from server_args.speculative_eagle_topk at init time instead of from spec_info at runtime. This avoids the dependency on SpecInput subtypes all defining topk, and is consistent with how the backend reads other config (pad_slot_id, device, etc.).

For ngram, speculative_eagle_topk is set to speculative_ngram_max_bfs_breadth in server_args, so tree attention branches execute correctly.

Fixes #20721

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[kpham-sgl]

The actual fix should be propagating speculative_eagle_topk to NgramVerifyInput. Its actually already set in

sglang/python/sglang/srt/server_args.py

Line 2987 in 9419453

self.speculative_eagle_topk = self.speculative_ngram_max_bfs_breadth

Conceptually, Ngram does build a spec tree (see https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/speculative/cpp_ngram/ngram.cpp#L257 and https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/speculative/cpp_ngram/ngram.cpp#L296)
The parameters that control the tree breadth and depth are

  # Tree breadth:                                                                                                                                                      
  --speculative-ngram-min-bfs-breadth (default: 1)                                                                                                                 
  --speculative-ngram-max-bfs-breadth (default: 10)                                                                                                                
                                                                                                                                                                     
  # Match window (tree depth):                                                                                                                                         
  --speculative-ngram-min-match-window-size (default: 1)                                                                                                           
  --speculative-ngram-max-match-window-size (default: 12)                                                                                                          
                                                                                                                                                                     
  # Other NGRAM params:                                                                                                                                                
  --speculative-ngram-branch-length (default: 18)                                                                                                                  
  --speculative-ngram-match-type (BFS or PROB, default: BFS)

[kpham-sgl]

To make it consistent with some other attention backend, the simpler fix is to read directly from server args

sglang/python/sglang/srt/layers/attention/flashattention_backend.py

Line 367 in 3867c64

self.topk = model_runner.server_args.speculative_eagle_topk or 0

sglang/python/sglang/srt/layers/attention/nsa_backend.py

Line 339 in 3867c64

self.topk = model_runner.server_args.speculative_eagle_topk or 0

Triton instantiate its own self.topk which can also be trace back to self.topk server_args.speculative_eagle_topk

@he-yufeng lmk what you think

[kpham-sgl]

/tag-and-rerun-ci

[hnyls2002]

/rerun-test test_hybrid_attn_backend.py test_ngram_speculative_decoding.py
(2 tries)

[github-actions]

✅ 1-gpu-h100 (1 test): View workflow run

cd test/ && python3 registered/spec/test_ngram_speculative_decoding.py

[github-actions]

✅ 1-gpu-h100 (2 tests): View workflow run

cd test/ && python3 registered/attention/test_hybrid_attn_backend.py
cd test/ && python3 registered/spec/test_ngram_speculative_decoding.py