Validate GGUF special token ids against vocab to prevent OOB panic

[pjdurden]

Summary

convert_gguf_to_hf_tokenizer reads eos/bos/unk token ids from GGUF
metadata and indexes straight into props.tokens without bounds-checking
(tokens[id as usize]). GGUF metadata is untrusted input, so a file whose
special-token ids exceed the vocab size panics the process during model load
instead of returning an error.

There are several such indexing sites: the GgufTokenizerConversion build
(eos/bos/unk) plus the per-token-id lookups in unigram_tokenizer and
bpe_tokenizer.

Fixes #2225.

Repro

A GGUF with a 1-token vocab and tokenizer.ggml.eos_token_id = 4294967295
(u32::MAX) aborts on load:

thread 'main' panicked at mistralrs-core/src/gguf/gguf_tokenizer.rs:
index out of bounds: the len is 1 but the index is 4294967295

eos_token_id is required for embedded-tokenizer GGUFs, so this path is always
hit when loading one.

Fix

Validate eos, bos, and unk against tokens.len() in
PropsGGUF::try_from, right after the struct is built. A single check before
any indexing covers every downstream lookup, and a malformed file now returns
a clear error:

GGUF `eos` token id 4294967295 is out of bounds for vocab size 1

Since eos is required and validated, vocab_size >= 1 holds, so the
unk.unwrap_or(0) -> tokens[0] path in unigram_tokenizer stays safe.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

Code Metrics Report

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Language              Files        Lines         Code     Comments       Blanks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 C Header                 23         4454         3116          790          548
 CSS                       3          282          252            6           24
 CUDA                    119        23620        19165         1704         2751
 Dockerfile                1           35           19            9            7
 HTML                      2           27           27            0            0
 JavaScript                3          577          562           12            3
 Jinja2                    7          694          656            5           33
 JSON                     27        15864        15861            0            3
 Makefile                  1           16           14            0            2
 MDX                      36         5872            0         4310         1562
 Metal Shading Lan|       37        14287        11284         1136         1867
 PowerShell                1          532          427           39           66
 Python                  143        11695         9677          473         1545
 Shell                     2          906          703          109           94
 Plain Text                3         3723            0         2413         1310
 TOML                     27         1334         1153           43          138
 TypeScript               11         1641         1404           66          171
 YAML                      3           25           23            2            0
─────────────────────────────────────────────────────────────────────────────────
 Jupyter Notebooks         3          122           83           23           16
 |- Markdown               1           60           30           22            8
 |- Python                 1          122          113            1            8
 (Total)                              304          226           46           32
─────────────────────────────────────────────────────────────────────────────────
 Markdown                266        11224            0         8347         2877
 |- BASH                  22          282          208           43           31
 |- Dockerfile             2            5            5            0            0
 |- JSON                   6          289          289            0            0
 |- PowerShell             1            1            1            0            0
 |- Python               129         7052         5852          310          890
 |- Rust                  62         3817         2838          385          594
 |- TOML                   7           77           65            0           12
 (Total)                            22747         9258         9085         4404
─────────────────────────────────────────────────────────────────────────────────
 Rust                    647       286535       254782         5892        25861
 |- Markdown             410         9762          452         8130         1180
 (Total)                           296297       255234        14022        27041
─────────────────────────────────────────────────────────────────────────────────
 Svelte                   19         1969         1826           51           92
 |- CSS                    1            4            4            0            0
 |- JavaScript            19          921          767           25          129
 (Total)                             2894         2597           76          221
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Total                  1384       407826       331658        34346        41822
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

[EricLBuehler]

Thank you @pjdurden!