Skip to content

Whisper: keep every timestamped segment in the transcript - #5554

Open
singhharsh1708 wants to merge 1 commit into
tenstorrent:mainfrom
singhharsh1708:fix/whisper-timestamps-join-segments
Open

singhharsh1708 wants to merge 1 commit into
tenstorrent:mainfrom
singhharsh1708:fix/whisper-timestamps-join-segments

Conversation

@singhharsh1708

Copy link
Copy Markdown

With return_timestamps=true, tt-metal's WhisperGenerator returns each batch item as a list of {"text", "start", "end"} segments instead of a string. TextUtils.extract_text (tt-media-server/utils/text_utils.py) kept only element [0] of that list, and returned an empty string when handed the segment list itself. At max_batch_size 1 the runner calls it on the generator tuple (whisper_runner.py, _format_non_streaming_result and the per-segment path), so each transcript or VAD chunk kept its first segment only; at max_batch_size 2 (T3K, Galaxy) every request goes through _execute_pipeline_batch, which passes each item's segment list, so the transcript came back empty.

extract_text now unwraps the generator tuple and the per-item list once, then joins every segment's text; start is the first segment's start and end the last segment's end. Strings, empty results and the streaming tuples are handled as before.

Measured on CPU with the server's extract_text unchanged on openai/whisper-large-v3 output (fp32, greedy, English) for 36 LibriSpeech clean windows of up to 15 s (1.3 segments on average): WER 13.9% through the batch-size-1 path (88% of the words kept) and 100% through the batch path, against 2.15% with all segments joined, which equals the WER without timestamps.

New tests/test_text_utils.py covers the batch-1 tuple, a per-item segment list, a single segment, plain strings and empty results; on main 3 of the 5 fail (first segment only, or empty), with this change tests/test_text_utils.py tests/test_whisper_chunk_metrics.py tests/test_whisper_stage_throughput.py: 35 passed. Not run on device. Per-segment boundaries are still not surfaced in the response beyond start and end.

@singhharsh1708
singhharsh1708 requested a review from a team as a code owner November 9, 2026 16:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant