fix(llg): honor tokenizer special flag in toktrie detok

[subin9]

Problem

Any model whose tokenizer has added tokens flagged special = false loses those tokens from the
decoded completion string, diverging from HuggingFace transformers. It's silent — no error, the
token ids are generated correctly; only the decode drops them.

Surfaced with PaddleOCR-VL, whose tables are emitted as OTSL added tokens <fcel> / <nl>
(special = false). They were dropped, collapsing a table into a run-on string of cell text with no
rows or columns:

greedy tokens = [<fcel>, A, <fcel>, B, <nl>, <fcel>, 1, <fcel>, 2, <nl>, </s>]

transformers  tokenizer.decode(..., skip_special_tokens=True) -> "<fcel>A<fcel>B<nl><fcel>1<fcel>2<nl>"
mistral.rs    engine completion (before this PR)              -> "AB12"   # <fcel>/<nl> stripped

This is general, not PaddleOCR-specific: any special = false <...>-shaped added token (e.g.
Qwen2.5-VL's <tool_call>) is affected.

Root cause (upstream heuristic)

toktrie_hf_tokenizers::ByteTokenizer::from_tokenizer marks a token special by a name-shape
heuristic that overrides the tokenizer's own flag (src/lib.rs:121-122 in the v1.4.0 this repo
resolves; :188 on current llguidance main):

if info.special || (info.content.starts_with("<") && info.content.ends_with(">")) {

So every <...>-shaped added token gets the SPECIAL_TOKEN_MARKER, and the completion detok —
tok_trie().decode_ext(tokens, include_special=false) — strips it. The heuristic is intentional
upstream (added in guidance-ai/llguidance#202 to mirror the transformers path for GGUF, where a
reliable flag is often absent), but it also overrides an explicit special = false. Filed
upstream: guidance-ai/llguidance#361. This PR is a correct local correction valid regardless of the
upstream outcome.

Precedent

Same class of bug was fixed narrowly in #1950 (merged), which enabled include_special during
decode so <think>/</think> delimiters appear — but only in think-tag mode. This PR fixes the
general case at the token-classification source instead of per-feature.

Fix

After from_tokenizer, strip the SPECIAL_TOKEN_MARKER from the token_bytes of added tokens the
tokenizer flags special = false, so they decode as content — matching
transformers.decode(skip_special_tokens=True). The existing loop that marks special = true
tokens is untouched, so EOS/BOS/image/chat-delimiter tokens are dropped exactly as before.
Decode-only: it never changes which token id is generated.

Verification (llg::tests::honors_tokenizer_special_flag, before/after on real tokenizers)

token	tokenizer flag	before	after
<fcel> (PaddleOCR-VL OTSL)	special=false	"" (dropped)	"<fcel>" (kept)
</s> (PaddleOCR-VL EOS)	special=true	""	""
<|IMAGE_END|> (PaddleOCR-VL)	special=true	""	""
<|im_start|> (Qwen2.5-VL chat delimiter)	special=true	""	"" (no leak)
<tool_call> (Qwen2.5-VL)	special=false	""	"<tool_call>" (now content, per HF)

special = true tokens stay dropped; only tokenizer-flagged special = false tokens survive.
clippy -p mistralrs-core --tests -- -D warnings clean. Two tests cover this:
honors_special_flag_inline_fixture builds a tiny byte-level tokenizer in-process (one
special=false <x>, one special=true <s>) and runs in CI with no external files;
honors_tokenizer_special_flag is the same before/after check against the real PaddleOCR-VL and
Qwen2.5-VL tokenizers (env-gated on local files, so it skips in CI when they're absent).

Tool-calling safety

The <tool_call> row changes a shipped feature's detok, so it was checked explicitly. Qwen
<tool_call> (special = false) surviving into content in a non-tool-calling generation does
not break tool-calling: seq.tool_call_state is Some only when tools are requested, and every
tool-parse entrypoint (prefix_status, complete_if_tool_call,
maybe_activate_continuation_grammar, finalize_for_response, the Gemma content matcher) is gated
on it — there is no ungated content-stream scan for tool markers. When tool_call_state.is_some()
detok already runs include_special=true, so <tool_call> is in the stream regardless of this fix;
tool sessions are byte-identical before/after. Non-tool sessions run no parser, so the token is
inert text (matching HF).

[github-actions]

Code Metrics Report

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Language              Files        Lines         Code     Comments       Blanks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 C Header                 23         4454         3116          790          548
 CSS                       3          282          252            6           24
 CUDA                    119        23681        19217         1704         2760
 Dockerfile                1           35           19            9            7
 HTML                      2           27           27            0            0
 JavaScript                3          577          562           12            3
 Jinja2                    7          694          656            5           33
 JSON                     27        15864        15861            0            3
 Makefile                  1           18           16            0            2
 MDX                      36         5901            0         4327         1574
 Metal Shading Lan|       37        14287        11284         1136         1867
 PowerShell                1          656          570           31           55
 Python                  151        12284        10191          484         1609
 Shell                     3         1062          843          117          102
 Plain Text               53        10687            0         9209         1478
 TOML                     28         1368         1189           39          140
 TypeScript               11         1641         1404           66          171
 YAML                      3           25           23            2            0
─────────────────────────────────────────────────────────────────────────────────
 Jupyter Notebooks         3          122           83           23           16
 |- Markdown               1           60           30           22            8
 |- Python                 1          122          113            1            8
 (Total)                              304          226           46           32
─────────────────────────────────────────────────────────────────────────────────
 Markdown                273        11726            0         8736         2990
 |- BASH                  23          295          217           46           32
 |- Dockerfile             2            5            5            0            0
 |- JSON                   6          289          289            0            0
 |- PowerShell             1            1            1            0            0
 |- Python               135         7239         6021          310          908
 |- Rust                  62         3820         2838          388          594
 |- TOML                   7           77           65            0           12
 (Total)                            23452         9436         9480         4536
─────────────────────────────────────────────────────────────────────────────────
 Rust                    674       299702       267081         5872        26749
 |- Markdown             413         9761          452         8126         1183
 (Total)                           309463       267533        13998        27932
─────────────────────────────────────────────────────────────────────────────────
 Svelte                   19         1969         1826           51           92
 |- CSS                    1            4            4            0            0
 |- JavaScript            19          921          767           25          129
 (Total)                             2894         2597           76          221
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Total                  1478       429656       345022        41537        43097
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━