Skip to content

spec: add spec metrics mean acceptance length and acceptance rate per position #24536

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ggerganov merged 5 commits into from
Jun 16, 2026
+57 −4
Merged

spec: add spec metrics mean acceptance length and acceptance rate per position #24536

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d5229ce
spec: add spec metrics mean acceptance length and acceptance per pos
ruixiang63 Jun 12, 2026
b6bc8f3
fix as suggestion
ruixiang63 Jun 15, 2026
a677a63
fix as suggestion
ruixiang63 Jun 15, 2026
6ef4d00
fix as suggestion
ruixiang63 Jun 15, 2026
c241629
fix as suggestions
ruixiang63 Jun 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 30 additions & 1 deletion common/speculative.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,8 @@ struct common_speculative_impl {
size_t n_gen_tokens = 0; // number of tokens generated by this implementation.
size_t n_acc_tokens = 0; // number of tokens accepted by the target model.

std::vector<size_t> n_acc_tokens_per_pos; // number of tokens accepted per draft position.

// TODO: track performance of most recent calls
const bool gen_perf = true; // whether to generate performance stats.

Expand Down Expand Up @@ -2059,6 +2061,15 @@ void common_speculative_accept(common_speculative * spec, llama_seq_id seq_id, u

{
common_time_meas tm(impl->t_accept_us, !impl->gen_perf);

if (impl->n_acc_tokens_per_pos.size() < n_accepted) {
impl->n_acc_tokens_per_pos.resize(n_accepted, 0);
}

for (size_t i = 0; i < n_accepted; ++i) {
impl->n_acc_tokens_per_pos[i]++;
}

if (n_accepted > 0) {
impl->n_acc_drafts++;
impl->n_acc_tokens += n_accepted;
Expand Down Expand Up @@ -2093,13 +2104,31 @@ void common_speculative_print_stats(const common_speculative * spec) {
str_perf = "";
}

LOG_INF("statistics %16s: #calls(b,g,a) = %4zu %6zu %6zu, #gen drafts = %6zu, #acc drafts = %5zu, #gen tokens = %6zu, #acc tokens = %5zu%s\n",
std::string str_stats;
if (impl->n_call_accept > 0) {
const double mean =
1.0 + (double) impl->n_acc_tokens / (double) impl->n_call_accept;

@ggerganov ggerganov Jun 15, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it correct to have 1.0 + here?

For example, if I see in the logs mean acc len = 3.0, should I use --spec-draft-n-max 3 or --spec-draft-n-max 2. I think it is the latter, which is a bit confusing.

@ruixiang63 ruixiang63 Jun 15, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the 1.0 + is intentional. It follows the conventional definition from vLLM: mean acceptance length = tokens committed per target forward pass, including the bonus token that's always produced even when no draft is accepted — so the range is [1, n_max+1], not [0, n_max]. see https://github.com/vllm-project/vllm/blob/b997071ec493765abbed990c65843ed05e4708a8/vllm/v1/spec_decode/metrics.py#L114
mean acc len = 3.0 corresponds to about 2 accepted draft tokens on average, and a total advancement of 3 tokens per target pass.

It is basically the speedup over normal decoding, which only produces 1 token per pass. The bonus token is included because the target always emits one even if every draft gets rejected, so AL = 1 means "no speedup".

That said, it's only a convention. If you think it reads better without the bonus token, I can report it without +1 instead.

@ruixiang63 ruixiang63 Jun 15, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More specifically, the +1 represents the one additional token produced by the target model: a recovered token if some draft token is rejected, or a bonus token if all draft tokens are accepted. So this metric is closer to "tokens advanced per speculative verification step".
For choosing n_max, the per-position acceptance rates are more preferred.

std::ostringstream tmp;
tmp << std::fixed << std::setprecision(3);
for (size_t i = 0; i < impl->n_acc_tokens_per_pos.size(); ++i) {
if (i > 0) {
tmp << ", ";
}
tmp << (double) impl->n_acc_tokens_per_pos[i] / (double) impl->n_call_accept;
}
std::ostringstream oss;
oss << std::fixed << std::setprecision(2) << mean;
str_stats = ", #mean acc len = " + oss.str() + ", #acc rate/pos = (" + tmp.str() + ")";
}

LOG_INF("statistics %16s: #calls(b,g,a) = %4zu %6zu %6zu, #gen drafts = %6zu, #acc drafts = %5zu, #gen tokens = %6zu, #acc tokens = %5zu%s%s\n",
common_speculative_type_to_str(impl->type).c_str(),
impl->n_call_begin, impl->n_call_draft, impl->n_call_accept,
impl->n_gen_drafts,
impl->n_acc_drafts,
impl->n_gen_tokens,
impl->n_acc_tokens,
str_stats.c_str(),
str_perf.c_str());
}
}
30 changes: 27 additions & 3 deletions tools/server/server-context.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -205,6 +205,8 @@ struct server_slot {
// Speculative decoding stats
int32_t n_draft_total = 0; // Total draft tokens generated
int32_t n_draft_accepted = 0; // Draft tokens actually accepted
int32_t n_draft_verif_steps = 0; // Total draft token verification steps by the target model
std::vector<int32_t> n_accepted_per_pos; // Accepted tokens per draft position

void reset() {
SLT_DBG(*this, "%s", "\n");
Expand All @@ -231,6 +233,8 @@ struct server_slot {
// clear speculative decoding stats
n_draft_total = 0;
n_draft_accepted = 0;
n_draft_verif_steps = 0;
n_accepted_per_pos.clear();

task_prev = std::move(task);
task.reset();
Expand Down Expand Up @@ -510,10 +514,22 @@ struct server_slot {
llama_perf_context(ctx_tgt).n_reused);

if (n_draft_total > 0) {
const float draft_ratio = (float) n_draft_accepted / n_draft_total;
const float draft_ratio = (float) n_draft_accepted / n_draft_total;
const double mean_acc_len = n_draft_verif_steps > 0 ? 1.0 + (double) n_draft_accepted / (double) n_draft_verif_steps : 1.0;

std::string acceptance_rates_per_pos;
if (n_draft_verif_steps > 0) {
for (size_t i = 0; i < n_accepted_per_pos.size(); ++i) {
if (i > 0) {
acceptance_rates_per_pos += ", ";
}
acceptance_rates_per_pos += string_format("%.3f", (double) n_accepted_per_pos[i] / (double) n_draft_verif_steps);
}
}

SLT_INF(*this,
"draft acceptance = %0.5f (%5d accepted / %5d generated)\n",
draft_ratio, n_draft_accepted, n_draft_total);
"draft acceptance = %0.5f (%5d accepted / %5d generated), mean acceptance length = %5.2f, acceptance rate per position = (%s)\n",
draft_ratio, n_draft_accepted, n_draft_total, mean_acc_len, acceptance_rates_per_pos.c_str());
}

common_speculative_print_stats(spec);
Expand Down Expand Up @@ -3503,6 +3519,14 @@ struct server_context_impl {

// update how many tokens out of those tested were accepted
slot.n_draft_accepted += ids.size() - 1;
slot.n_draft_verif_steps += 1;

if (slot.n_accepted_per_pos.empty()) {
slot.n_accepted_per_pos.resize(common_speculative_n_max(&params_base.speculative), 0);
}
for (size_t i = 0; i < ids.size() - 1 && i < slot.n_accepted_per_pos.size(); ++i) {
slot.n_accepted_per_pos[i]++;
}

// add accepted tokens to the prompt
slot.prompt.tokens.keep_first(slot.prompt.n_tokens() - n_draft);
Expand Down
Loading