Joyo Kanji Yomi Benchmark

A benchmark toolkit for evaluating Japanese Text-to-Speech (TTS) pronunciation accuracy at the kanji level using Kana-CER.

Provides 13,095 native-speaker-verified test samples covering 2,136 Joyo kanji and their 4,378 readings, with sentence and kanji level katakana annotations.

Features

• Full Joyo Kanji coverage: 2,136 kanji × 4,378 readings × 3 sentences, covering every kanji-reading pair in the official Regular-use Kanji List
• Kanji-level error attribution: Each sentence annotates the target kanji's reading with <> delimiters, enabling per-reading error analysis to pinpoint exactly which readings a TTS system struggles with
• Kana-CER metric: Evaluates pronunciation in katakana space, eliminating spurious errors from Japanese orthographic variation (e.g., 「行う」vs「おこなう」vs「行なう」)
• Bundled Kana-ASR model: Ships with a fine-tuned Whisper model (sbintuitions/kana-whisper) that transcribes speech directly into katakana sequences
• Ready-to-run pipeline: Plug in synthesized audio and get per-sample and aggregated results in one command

Installation

# Clone the repository
git clone https://github.com/sbintuitions/Joyo-Kanji-Yomi-Benchmark
cd JoyoKanji-Yomi-Benchmark

# (Optional) Create and activate a virtual environment
python -m venv venv
source venv/bin/activate

# Install dependencies
pip install -e .

Quick Start

Each sample in the benchmark data has the following format:

{
  "key": "精_セイ_0",
  "normalized_text": "その時計は非常に精密な作りになっている。",
  "normalized_pron": "ソノトケーワヒジョーニ<セー>ミツナツクリニナッテール。"
}

1. Synthesize audio from the normalized_text field using your TTS system.
2. Name each audio file as {key}.wav and place them under a synthesized_audio/ subdirectory. You can create a separate directory for each TTS system:

results/
├── my_tts_model_A/
│   └── synthesized_audio/
│       ├── 精_セイ_0.wav
│       ├── 精_セイ_1.wav
│       └── ...
└── my_tts_model_B/
    └── synthesized_audio/
        ├── 精_セイ_0.wav
        ├── 精_セイ_1.wav
        └── ...

3. Run evaluation by pointing --dataset-dir to the directory containing synthesized_audio/:

python3 eval_dataset.py --dataset-dir results/my_tts_model_A
python3 eval_dataset.py --dataset-dir results/my_tts_model_B

The benchmark dataset (sbintuitions/joyo-kanji-yomi-benchmark) will be automatically downloaded on first run if not present. Results will be saved to eval_results/ under the dataset-dir directory. See summary.txt for the aggregated metrics.

Command-Line Options & Output

Arguments

The aruguments of eval_dataset.py.

Argument	Required	Default	Description
--dataset-dir	Yes	—	Directory containing synthesized_audio/
--source-jsonl	No	data/common_kanji_source.jsonl	Ground truth file (auto-downloads if missing)
--kana-model	No	sbintuitions/kana-whisper	ASR model for Kana-CER computation
--text-model	No	openai/whisper-large-v3-turbo	ASR model for Standard CER computation
--result-dir	No	{dataset-dir}/eval_results	Output directory
--skip-asr	No	false	Skip ASR if transcription files already exist

Output Files

eval_results/
├── transcription.kana.jsonl        # ASR output from Kana-ASR model
├── transcription.text.jsonl        # ASR output from Text-ASR model
├── samples.kana_cer.jsonl          # Per-sample Kana-CER
├── samples.sent_kana_cer.jsonl     # Per-sample Sent-Kana-CER
├── samples.cer.jsonl               # Per-sample Standard CER
├── stats.kana_cer.json             # Aggregated Kana-CER stats
├── stats.sent_kana_cer.json        # Aggregated Sent-Kana-CER stats
├── stats.cer.json                  # Aggregated Standard CER stats
└── summary.txt                     # Human-readable summary

Metrics

The toolkit computes three types of CER metrics:

• Kana-CER: Measures reading selection correctness for the specific target kanji, using the <>-delimited span in the reference annotation
• Sent-Kana-CER: Measures overall pronunciation correctness across the entire sentence
• Standard CER: Standard CER via Whisper, provided as a reference metric for comparison with prior work

To evaluate on your own dataset instead of the Joyo Kanji Yomi Benchmark, provide a custom JSONL file via --source-jsonl. Each entry must contain a normalized_pron field with <>-tagged spans to indicate the target kanji's reading (e.g., カレラワ<ケ>ッキョセーカツ). Kana-CER is computed only on the tagged spans; Sent-Kana-CER and Standard CER are computed on the full sentence regardless.

Citation

If you use the Joyo Kanji Yomi Benchmark in your research, please cite:

@misc{liu2026sarashina22ttstacklingkanjipolyphony,
      title={Sarashina2.2-TTS: Tackling Kanji Polyphony in Japanese Speech Generation via Data Scaling and Targeted Data Synthesis}, 
      author={Lianbo Liu and Shiao Zhu and Kai Washizaki and Reo Yoneyama and Haesung Jeon and Mengjie Zhao and Yusuke Fujita and Hao Shi and Nao Yoshida and Yuan Gao and Roman Koshkin and Yukiya Hono and Yui Sudo},
      year={2026},
      eprint={2606.25369},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2606.25369}, 
}