[ydshieh]

Problem

In TokenizersBackend.convert_to_native_format(), when a tokenizer has a custom __init__ (the elif branch), tokenizer.json was parsed twice:

1. TokenizerFast.from_file(fast_tokenizer_file) — full Rust parse including the entire vocab (~0.6s for large tokenizers like CohereTokenizer with 256k entries) — used only to extract post_processor, padding, truncation
2. json.load(tokenizer_handle) — Python parse of the same file — to extract vocab and merges

Fix

Read the file once with json.load, then build a minimal tokenizer (same structure but empty vocab/merges) via TokenizerFast.from_str to cheaply obtain the Rust post_processor/padding/truncation objects — without parsing the full vocab.

Benchmark

Measured over 300 runs of AyaVisionProcessor.from_pretrained("local") (AyaVisionProcessor uses CohereTokenizer which has a custom __init__):

With fix    : avg=1.151s  min=0.987s  max=1.422s
Without fix : avg=1.545s  min=1.311s  max=2.513s
Speedup     : 1.34x  (~0.4s saved per load on average)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ydshieh]

Benchmark

Script used (bench.py):

import time
import os
import sys
from transformers import (
    CohereTokenizer,
    LlamaTokenizer,
    GemmaTokenizer,
)

N = 100

TOKENIZERS = {
    "CohereTokenizer":     (CohereTokenizer,      "hf-internal-testing/namespace-CohereForAI-repo_name_aya-vision-8b"),
    "LlamaTokenizer":      (LlamaTokenizer,       "huggyllama/llama-7b"),
    "GemmaTokenizer":      (GemmaTokenizer,       "google/gemma-7b"),
}

if len(sys.argv) < 2:
    print(f"Usage: python bench.py <TokenizerClassName>")
    print(f"Available: {', '.join(TOKENIZERS)}")
    sys.exit(1)

name = sys.argv[1]
if name not in TOKENIZERS:
    print(f"Unknown tokenizer: {name}. Available: {', '.join(TOKENIZERS)}")
    sys.exit(1)

cls, hub_id = TOKENIZERS[name]
local_path = f"local_bench_{name}"

# Download from hub and save locally if not cached
if not os.path.isdir(local_path):
    print(f"Downloading {name} from {hub_id}...")
    obj = cls.from_pretrained(hub_id)
    obj.save_pretrained(local_path)
    print(f"Saved to {local_path}")

# Run benchmark
times = []
for _ in range(N):
    t0 = time.perf_counter()
    cls.from_pretrained(local_path)
    times.append(time.perf_counter() - t0)

total = sum(times)
print(f"{name}  N={N}  total={total:.3f}s  avg={total/N:.3f}s  min={min(times):.3f}s  max={max(times):.3f}s")

Run with: python bench.py <TokenizerClassName> (with fix at HEAD, without fix at 0567834cbc).

Results (N=100)

Tokenizer	Vocab size	With fix	Without fix	Speedup
CohereTokenizer	256k	• avg=1.318s • min=1.235s • max=1.603s	• avg=1.863s • min=1.714s • max=2.172s	1.41x
LlamaTokenizer	32k	• avg=0.222s • min=0.180s • max=0.312s	• avg=0.305s • min=0.259s • max=0.450s	1.37x
GemmaTokenizer	256k	• avg=1.708s • min=1.505s • max=1.905s	• avg=2.441s • min=2.204s • max=2.677s	1.43x

[tarekziade]

like this one can be initialized out of this if block since it's always done

[tarekziade]

this list could be a constant alongside MODEL_TO_TRAINER_MAPPING

[ydshieh]

I will change it to model_type != "Unigram"

[ArthurZucker]

Good catch!

[ArthurZucker]

for me this should be a feature request for tokenizers to allows post_processor.from_str().
Its kinda hacky but I guess should work well!

[ydshieh]

@tarekziade applied your suggestions, thx (force pushed after git rebase)