Skip to content

About

Voice benchmark harness: word error rate on LibriSpeech with documented normalization, reported next to nearest-rank p50/p95 latency over warm runs.

Resources

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

speko-voicebench

A small, reproducible STT benchmark: two providers, the same 15 English clips, measuring latency (p50/p95) and accuracy (WER). Runs from a clean clone with no API key required.

Quickstart (no key needed)

git clone https://github.com/Kayvan-Zahiri/speko-voicebench.git speko-voicebench && cd speko-voicebench
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python bench.py

First run downloads the faster-whisper small.en model (~484 MB, one time). The 15 audio clips and the Deepgram response fixtures are committed, so bench.py reproduces the numbers below with no key. To re-measure Deepgram live, put DEEPGRAM_API_KEY=... in .env (see .env.example) and re-run.

Results

Captured 2026-07-10. 15 LibriSpeech dev-clean clips, 3 warm runs each (n=45 latency samples/provider).

Provider WER latency p50 latency p95
faster-whisper small.en (int8, local CPU) 0.084 1207 ms 3495 ms
Deepgram nova-2 (hosted) 0.045 234 ms 560 ms

Providers & pinned versions

  • Local: faster-whisper==1.2.1, model small.en, compute_type=int8, CPU, beam_size=5.
  • Hosted: Deepgram nova-2 via deepgram-sdk==7.4.0, smart_format=true, language=en.
  • WER via jiwer==4.0.0; dataset via datasets==5.0.0. Full pins in requirements.txt.

Method

  • Inputs: 15 LibriSpeech dev-clean clips (data/fixtures/), 145 s total, 2.2–29.4 s each. Identical clips for both providers.
  • Latency: raw end-to-end (audio in -> transcript out), 3 warm runs/clip. Local loads the model once outside the timer (warm inference, not cold start). Percentiles are nearest-rank (ceil(p/100 * N)).
  • Accuracy: corpus WER (total edits / total reference words) via jiwer. Normalization is intentionally transparent -- lowercase, strip punctuation, collapse whitespace (metrics.normalize), applied identically to reference and hypothesis.
  • Keyless reproduction: Deepgram transcripts + measured latencies are recorded to recordings/*.json and committed; the harness replays them when no key is set. A replayed latency is a measurement from the 2026-07-10 capture on my connection, not a live number.

What this does NOT measure

  • Latency isn't hardware-controlled. Local runs on my laptop CPU with no network; Deepgram is a cloud call that includes the network round-trip. Deepgram being faster reflects cloud hardware plus my connection, not that its model is intrinsically faster -- on a GPU, or a worse network, this could flip.
  • Raw latency mixes clip length with model speed. Clips span 2.2-29.4 s, so the p95 tail is driven mostly by the longest clips, not by stalls -- that's why Whisper's p95 is ~3x its p50. A length-normalized metric (real-time factor) would isolate speed; I report raw because it's what a caller actually waits.
  • WER is inflated by formatting, not just errors. Whisper writes "Mr." and digits; Deepgram and LibriSpeech spell out "Mister" and numbers. My transparent normalization doesn't reconcile those, so Whisper is penalized for a convention it got right. I would not conclude Whisper is less accurate from this -- a normalizer like Whisper's own would shrink the gap.
  • The sample is tiny. 15 clips, 45 latency runs. p95 rides on ~2 tail samples and WER on 15 utterances, so both carry wide uncertainty -- I wouldn't treat a small WER difference as significant.
  • Clean read speech only. LibriSpeech dev-clean is studio-quality, single-speaker audio. Real call audio -- noise, accents, cross-talk, jargon, phone codecs -- would push WER far higher for both providers and could reorder them. Nothing here predicts production voice performance.

Reproduce / regenerate

  • Re-run the benchmark: python bench.py (writes results.json).
  • Regenerate the dataset: python data/get_data.py.
  • Re-capture Deepgram live: set DEEPGRAM_API_KEY in .env, then python bench.py.

About

Voice benchmark harness: word error rate on LibriSpeech with documented normalization, reported next to nearest-rank p50/p95 latency over warm runs.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages