Summary

ByteTokenizer::from_tokenizer classifies any added token shaped like <...> as special via a
name heuristic, even when the source tokenizers object explicitly flags that token
special = false. On decode/decode_ext(..., include_special=false) those tokens are then
stripped, which diverges from HuggingFace transformers
tokenizer.decode(ids, skip_special_tokens=True) (which honors the per-token flag and keeps them).

This is a narrow edge case for content-carrying markup tokens that happen to be <...>-shaped.

Where

toktrie_hf_tokenizers/src/lib.rs:188 (current main, crate v1.7.6); same heuristic at
src/lib.rs:121-122 in v1.4.0 (the version resolved by downstream mistral.rs):

// we treat all added tokens of the form <...> as special tokens
if info.special || (info.content.starts_with("<") && info.content.ends_with(">")) {

I understand the <...> rule is intentional — it was added deliberately in #202 to make the
llamacpp path match the transformers path, since GGUF metadata often lacks a reliable special
flag. The gap is only in the second half of the ||: when the tokenizer does provide an
explicit special = false, the heuristic still overrides it.

Repro

A tokenizer with an added token flagged special = false whose content is <...>-shaped — e.g.
PaddleOCR-VL's OTSL table tokens <fcel> / <nl>, or Qwen2.5-VL's <tool_call>. These are
content, not control tokens:

transformers  tokenizer.decode([<fcel>, A], skip_special_tokens=True) -> "<fcel>A"   # honors special=false
tokenizers    (Rust crate)   .decode([<fcel>, A], true)               -> "<fcel>A"   # honors special=false
llguidance    tok_trie.decode_ext([<fcel>, A], include_special=false) -> "A"         # heuristic dropped <fcel>

For OCR this collapses a structured table (<fcel>/<nl> mark cells/rows) into a run-on string
with the structure gone. The token ids are fine — only the decode drops them.

Proposed minimal change

Only apply the <...> shape heuristic when the tokenizer has not given an explicit special
flag for that token — i.e. let an explicit special = false win over the name shape. The
info.special branch and the GGUF/llamacpp fallback (where no reliable flag exists) stay exactly as
today; this only stops the shape heuristic from overriding a flag the tokenizer author set on
purpose.

Happy to send a PR if that direction sounds right, or adjust if you'd rather gate it behind an
option to preserve current behavior for callers that rely on it.

Context

Surfaced while adding a document-OCR model downstream in mistral.rs; we've applied a local
correction there (honor the tokenizer's own special flag after from_tokenizer) and are filing
this so the root behavior can be considered upstream. Related prior art in this repo: #202 (origin
of the heuristic), #177 (preserve special tokens in decode).