Repository navigation
Audio: decode 24-bit PCM and WAVE_FORMAT_EXTENSIBLE float WAV uploads correctly - #5556
Open
singhharsh1708 wants to merge 1 commit into
Open
singhharsh1708 wants to merge 1 commit into
singhharsh1708 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
/v1/audio/transcriptionsdecodes WAV uploads inAudioManager._decode_wav_file(tt-media-server/utils/audio_manager.py). Two sample formats come out wrong:24-bit PCM pads each 3-byte little-endian sample with a trailing zero byte and shifts the int32 right by 8, which computes floor(unsigned value / 256) before dividing by 2^23: positive samples land about 256 times too quiet and negative samples wrap to positive values. The decoder now pads with a leading zero byte and shifts, vectorised, so the sign survives.
32-bit samples are read as float only when the format tag is 3. Float WAVs with a WAVE_FORMAT_EXTENSIBLE header (tag 0xFFFE, which ffmpeg and sox write for float audio above 48 kHz or with more than two channels, and what soundfile's WAVEX writes) were read as int32 PCM. The decoder now takes the format tag from the SubFormat GUID when the header is extensible; 16-bit and 24-bit extensible PCM were already decoded by bit depth and are unchanged.
Measured on CPU:
openai/whisper-large-v3on the 73 LibriSpeech clean clips ofhf-internal-testing/librispeech_asr_dummy, saved as 16 kHz WAV and decoded with a line-for-line copy of this function (24-bit) or the function itself (extensible float), gives WER 37.2% for 24-bit PCM and 29.5% for extensible float, against 2.65% when the same bytes are decoded with soundfile. The 24-bit waveform has SNR -0.3 dB against the source and a peak of 0.008 where the source peaks at 0.526.Tests added to
tests/test_audio_manager.py: a 24-bit WAV written with thewavemodule and an extensible float WAV with a hand-built header, both checked against the source samples to one 24-bit step, plus a round trip against soundfile for both formats (importorskip, so it is skipped where soundfile is absent). On main all four fail (max abs error 1.0 and 0.59 on the hand-built files); with this changetests/test_audio_manager.pyhas 47 passed and the same one pre-existing failure as main on this machine (test_cpu_seconds_measures_a_real_subprocess, which reads /proc). Not run on device.