Repository navigation
Whisper: keep every timestamped segment in the transcript - #5554
Open
singhharsh1708 wants to merge 1 commit into
Open
singhharsh1708 wants to merge 1 commit into
singhharsh1708 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With
return_timestamps=true, tt-metal'sWhisperGeneratorreturns each batch item as a list of{"text", "start", "end"}segments instead of a string.TextUtils.extract_text(tt-media-server/utils/text_utils.py) kept only element[0]of that list, and returned an empty string when handed the segment list itself. Atmax_batch_size1 the runner calls it on the generator tuple (whisper_runner.py,_format_non_streaming_resultand the per-segment path), so each transcript or VAD chunk kept its first segment only; atmax_batch_size2 (T3K, Galaxy) every request goes through_execute_pipeline_batch, which passes each item's segment list, so the transcript came back empty.extract_textnow unwraps the generator tuple and the per-item list once, then joins every segment's text;startis the first segment's start andendthe last segment's end. Strings, empty results and the streaming tuples are handled as before.Measured on CPU with the server's
extract_textunchanged onopenai/whisper-large-v3output (fp32, greedy, English) for 36 LibriSpeech clean windows of up to 15 s (1.3 segments on average): WER 13.9% through the batch-size-1 path (88% of the words kept) and 100% through the batch path, against 2.15% with all segments joined, which equals the WER without timestamps.New
tests/test_text_utils.pycovers the batch-1 tuple, a per-item segment list, a single segment, plain strings and empty results; on main 3 of the 5 fail (first segment only, or empty), with this changetests/test_text_utils.py tests/test_whisper_chunk_metrics.py tests/test_whisper_stage_throughput.py: 35 passed. Not run on device. Per-segment boundaries are still not surfaced in the response beyondstartandend.