fix(core): reject out-of-range raw token ids in token requests

[blasrodri]

Problem

CompletionTokens and EmbeddingTokens requests carry raw token ids straight from the client (e.g. POST /v1/embeddings with {"input": [[...]]}). These ids were never range-checked before being handed to the model.

An id >= vocab_size indexes past the embedding table. On CPU this returns a graceful error, but on a CUDA release build the bounds assertion is compiled out (NDEBUG), so the gather performs an out-of-bounds device memory read. A single unauthenticated request such as {"model":"default","input":[4294967294]} can therefore hit undefined behavior in the engine instead of a clean rejection.

Fix

In the shared CompletionTokens | EmbeddingTokens arm of Engine::handle_request, validate every id against tokenizer.get_vocab_size(true) and return a ValidationError (HTTP 422) before the tokens reach the model. This mirrors the tokenizer-missing validation already present a few lines above in the same arm.

Covers both raw-token entry points (embeddings and completions) since they share the arm.

Tests

The range check is extracted into a pure first_out_of_range_token helper with unit tests covering: ids below vocab size, the max valid id (vocab_size - 1), an id equal to vocab_size (rejected at the boundary), an out-of-range id reported back to the caller, and the empty-input case.

[github-actions]

Code Metrics Report

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Language              Files        Lines         Code     Comments       Blanks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 C Header                 23         4454         3116          790          548
 CSS                       3          282          252            6           24
 CUDA                    119        23681        19217         1704         2760
 Dockerfile                1           35           19            9            7
 HTML                      2           27           27            0            0
 JavaScript                3          577          562           12            3
 Jinja2                    7          694          656            5           33
 JSON                     27        15864        15861            0            3
 Makefile                  1           18           16            0            2
 MDX                      36         5901            0         4327         1574
 Metal Shading Lan|       37        14287        11284         1136         1867
 PowerShell                1          657          571           31           55
 Python                  151        12284        10191          484         1609
 Shell                     3         1068          849          116          103
 Plain Text               53        10687            0         9209         1478
 TOML                     28         1368         1189           39          140
 TypeScript               11         1649         1410           66          173
 YAML                      3           25           23            2            0
─────────────────────────────────────────────────────────────────────────────────
 Jupyter Notebooks         3          122           83           23           16
 |- Markdown               1           60           30           22            8
 |- Python                 1          122          113            1            8
 (Total)                              304          226           46           32
─────────────────────────────────────────────────────────────────────────────────
 Markdown                273        11726            0         8736         2990
 |- BASH                  23          295          217           46           32
 |- Dockerfile             2            5            5            0            0
 |- JSON                   6          289          289            0            0
 |- PowerShell             1            1            1            0            0
 |- Python               135         7239         6021          310          908
 |- Rust                  62         3820         2838          388          594
 |- TOML                   7           77           65            0           12
 (Total)                            23452         9436         9480         4536
─────────────────────────────────────────────────────────────────────────────────
 Rust                    674       300652       267964         5856        26832
 |- Markdown             413         9761          452         8126         1183
 (Total)                           310413       268416        13982        28015
─────────────────────────────────────────────────────────────────────────────────
 Svelte                   19         1969         1826           51           92
 |- CSS                    1            4            4            0            0
 |- JavaScript            19          921          767           25          129
 (Total)                             2894         2597           76          221
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Total                  1478       430621       345918        41520        43183
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━