A small, reproducible STT benchmark: two providers, the same 15 English clips, measuring latency (p50/p95) and accuracy (WER). Runs from a clean clone with no API key required.
git clone https://github.com/Kayvan-Zahiri/speko-voicebench.git speko-voicebench && cd speko-voicebench
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
python bench.pyFirst run downloads the faster-whisper small.en model (~484 MB, one time). The
15 audio clips and the Deepgram response fixtures are committed, so bench.py
reproduces the numbers below with no key. To re-measure Deepgram live, put
DEEPGRAM_API_KEY=... in .env (see .env.example) and re-run.
Captured 2026-07-10. 15 LibriSpeech dev-clean clips, 3 warm runs each (n=45 latency samples/provider).
| Provider | WER | latency p50 | latency p95 |
|---|---|---|---|
faster-whisper small.en (int8, local CPU) |
0.084 | 1207 ms | 3495 ms |
Deepgram nova-2 (hosted) |
0.045 | 234 ms | 560 ms |
- Local:
faster-whisper==1.2.1, modelsmall.en,compute_type=int8, CPU,beam_size=5. - Hosted: Deepgram
nova-2viadeepgram-sdk==7.4.0,smart_format=true,language=en. - WER via
jiwer==4.0.0; dataset viadatasets==5.0.0. Full pins inrequirements.txt.
- Inputs: 15 LibriSpeech dev-clean clips (
data/fixtures/), 145 s total, 2.2–29.4 s each. Identical clips for both providers. - Latency: raw end-to-end (audio in -> transcript out), 3 warm runs/clip. Local loads the model once outside the timer (warm inference, not cold start). Percentiles are nearest-rank (
ceil(p/100 * N)). - Accuracy: corpus WER (total edits / total reference words) via jiwer. Normalization is intentionally transparent -- lowercase, strip punctuation, collapse whitespace (
metrics.normalize), applied identically to reference and hypothesis. - Keyless reproduction: Deepgram transcripts + measured latencies are recorded to
recordings/*.jsonand committed; the harness replays them when no key is set. A replayed latency is a measurement from the 2026-07-10 capture on my connection, not a live number.
- Latency isn't hardware-controlled. Local runs on my laptop CPU with no network; Deepgram is a cloud call that includes the network round-trip. Deepgram being faster reflects cloud hardware plus my connection, not that its model is intrinsically faster -- on a GPU, or a worse network, this could flip.
- Raw latency mixes clip length with model speed. Clips span 2.2-29.4 s, so the p95 tail is driven mostly by the longest clips, not by stalls -- that's why Whisper's p95 is ~3x its p50. A length-normalized metric (real-time factor) would isolate speed; I report raw because it's what a caller actually waits.
- WER is inflated by formatting, not just errors. Whisper writes "Mr." and digits; Deepgram and LibriSpeech spell out "Mister" and numbers. My transparent normalization doesn't reconcile those, so Whisper is penalized for a convention it got right. I would not conclude Whisper is less accurate from this -- a normalizer like Whisper's own would shrink the gap.
- The sample is tiny. 15 clips, 45 latency runs. p95 rides on ~2 tail samples and WER on 15 utterances, so both carry wide uncertainty -- I wouldn't treat a small WER difference as significant.
- Clean read speech only. LibriSpeech dev-clean is studio-quality, single-speaker audio. Real call audio -- noise, accents, cross-talk, jargon, phone codecs -- would push WER far higher for both providers and could reorder them. Nothing here predicts production voice performance.
- Re-run the benchmark:
python bench.py(writesresults.json). - Regenerate the dataset:
python data/get_data.py. - Re-capture Deepgram live: set
DEEPGRAM_API_KEYin.env, thenpython bench.py.