Feature Request: Add MTP / speculative decoding support to llama-bench for PR #22673

[Ajay9o9]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

The draft MTP PR #22673 adds speculative decoding / MTP support to:

llama-cli
llama-server

using flags such as:
--spec-type mtp
--spec-draft-n-max 3
--spec-draft-ngl 99

These flags work correctly in:

llama-cli
llama-server

(have only tested on these two)

but are unrecognized in llama-bench
Example:

./build/bin/llama-bench \
  -m model.gguf \
  --spec-type mtp
 -- spec-draft-n-max 3,4,5
 -- spec-draft-ngl 99,100

returns:

error: invalid parameter for argument: --spec-type

Motivation

Currently MTP benchmarking requires custom scripts or llama-server setups since speculative decoding flags are unavailable in llama-bench.

Adding support directly to llama-bench would make it much easier for users to quickly test and compare speculative decoding configurations such as different n-max values, context depths, throughput scaling, and VRAM tradeoffs using the built-in benchmarking tooling.

Possible Implementation

I did some digging and it looks like llama-bench currently uses its own parser / benchmark loop instead of the shared common argument infrastructure used by llama-cli and llama-server, which is likely why the speculative decoding flags are currently unavailable there.

[ruixiang63]

llama-bench measures performance with synthetic tokens. That's fine for regular models, but it doesn't work well for speculative decoding: SD speedups depend heavily on the actual prompts, the task category, the serving settings, and so on, and random tokens can't capture realistic draft acceptance.

For SD specifically, you may want to check this PR (#23869), which uses SPEED-Bench (an industry-standard speculative decoding benchmark) to measure SD performance end-to-end against llama-server.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.