whisper-base_timestamped broken with chunk_length_s=30

[jozefchutka]

System Info

transformers.js: 3.6.1

Environment/Platform

• Website/web-app
• Browser extension
• Server-side (e.g., Node.js, Deno, Bun)
• Desktop app (e.g., Electron)
• Other (e.g., VSCode extension)

Description

Using chunk_length_s=30 and onnx-community/whisper-base_timestamped produces broken timestamsp

Reproduction

Run the following code and notice the output in console log using the attached src.pcm (in .zip)

<script type="module">
const { env, pipeline } = await import("https://cdn.jsdelivr.net/npm/@huggingface/transformers@3.6.1/dist/transformers.min.js");
env.allowLocalModels = false;

const buffer = await (await fetch("src.pcm")).arrayBuffer();
const audio = new Float32Array(buffer);

const pipe = await pipeline("automatic-speech-recognition",
	"onnx-community/whisper-base_timestamped",
	{dtype:{encoder_model:"fp32", decoder_model_merged:"q4"},
	device:"webgpu"});

const result = await pipe(audio, {
	chunk_length_s: 30,
	stride_length_s: 5,
	return_timestamps: "word",
	language: "en"});

console.log(result.chunks.map(chunk => `${chunk.timestamp[0]} -> ${chunk.timestamp[1]} ${chunk.text}`))
</script>

it prints:

"29.98 -> 29.98  every",
"29.98 -> 29.98  day",
"29.98 -> 29.98  style."

Timestamps are invalid and there is also far more speaking.

Changing chunk_length_s to 29 fixes the issue and produces rather valid output:

"0 -> 0.42  everyday",
"0.42 -> 0.86  style.",
"1.38 -> 1.56  - True",
"1.56 -> 2  classic",
"2 -> 2.5  delivers",
"2.5 -> 3.02  premium",
"3.02 -> 3.54  essentials",
"3.54 -> 3.84  built",
"3.84 -> 4.08  for",
"4.08 -> 4.42  real",
"4.42 -> 4.9  life.",
"5.4 -> 5.64  Grab",
"5.64 -> 6.04  yours",
"6.04 -> 6.36  at",
"6.36 -> 6.86  Target,",
"7.3 -> 7.78  Costco,",
"8.28 -> 8.3  or",
"8.3 -> 8.5  head",
"8.5 -> 8.7  to",
"8.7 -> 9.34  TrueClassic",
"9.34 -> 10.04 .com",
"10.04 -> 12.28 /p4p.",
"12.86 -> 13.08  Get",
"13.08 -> 13.38  hooked",
"13.38 -> 13.52  up",
"13.52 -> 13.94  today.",
"14.16 -> 14.24  Now",
"14.24 -> 14.46  before",
"14.46 -> 14.62  we",
"14.62 -> 14.82  go,",
"15.1 -> 15.24  just",
"15.24 -> 15.42  wanna",
"15.42 -> 15.56  give",
"15.56 -> 15.68  a",
"15.68 -> 15.86  big",
"15.86 -> 16.1  shout",
"16.1 -> 16.28  out",
"16.28 -> 16.76  to",
"16.76 -> 16.9  the",
"16.9 -> 17.52  CEO",
"17.52 -> 17.86  and",
"17.86 -> 18.32  founder,",
"18.48 -> 18.6  Ryan",
"18.6 -> 18.92  Frouder,",
"18.98 -> 19.06  for",
"19.06 -> 19.22  coming",
"19.22 -> 19.36  on",
"19.36 -> 19.5  our",
"19.5 -> 19.76  show",
"19.76 -> 20.4  and",
"20.4 -> 20.6  just",
"20.6 -> 20.86  showing",
"20.86 -> 21.08  some",
"21.08 -> 21.28  love.",
"21.46 -> 21.62  Now,",
"21.9 -> 22.36  let's",
"22.36 -> 22.46  get",
"22.46 -> 22.7  back",
"22.7 -> 23.06  to",
"23.06 -> 23.22  the",
"23.22 -> 23.54  episode",
"24.32 -> 24.44  I",
"24.44 -> 24.6  mean",
"24.6 -> 25.4  like",
"25.4 -> 25.52  I",
"25.52 -> 25.68  said",
"25.68 -> 26.12  we're",
"26.12 -> 26.34  going",
"26.34 -> 26.58  through",
"26.58 -> 26.82  that",
"26.82 -> 27.34  we're",
"27.34 -> 27.56  losing",
"27.56 -> 28.02  stars",
"28.02 -> 29.26  and",
"29.26 -> 29.38  then",
"29.38 -> 29.56  we",
"29.56 -> 29.84  kind",
"29.84 -> 29.98  of"

Why is 30 broken in this case? Is 29 safer in all cases or is it just coincidence?

[xenova]

Hmm very strange 🤔 Do you know if this happens with the python library too?

[jozefchutka]

Following python produces correct output with timestamps. @xenova can you please look into what is wrong with js verion ?

!pip install -q transformers soundfile

import torch
import requests
import zipfile
import io
import numpy as np
import soundfile as sf
from transformers import pipeline

pipeline = pipeline(
    task="automatic-speech-recognition",
    model="openai/whisper-base",
    torch_dtype=torch.float16,
    device=0
)

response =  requests.get("https://github.com/user-attachments/files/21100555/src.zip")
zip_file = zipfile.ZipFile(io.BytesIO(response.content))
with zip_file.open("src.pcm") as src_pcm:
  audio = np.frombuffer(src_pcm.read(), dtype=np.float32)

sf.write("output.wav", audio, samplerate=16000, subtype="PCM_16")

pipeline(
    "output.wav", 
    return_timestamps="word"
)

'chunks': [{'text': ' every', 'timestamp': (0.0, 0.28)},
  {'text': ' day', 'timestamp': (0.28, 0.5)},
  {'text': ' style.', 'timestamp': (0.5, 1.26)},
  {'text': ' True', 'timestamp': (1.26, 1.56)},
  {'text': ' classic', 'timestamp': (1.56, 1.9)},
  {'text': ' delivers', 'timestamp': (1.9, 2.48)},
  {'text': ' premium', 'timestamp': (2.48, 3.02)},
  {'text': ' essentials', 'timestamp': (3.02, 3.54)},
  {'text': ' built', 'timestamp': (3.54, 3.84)},
  {'text': ' for', 'timestamp': (3.84, 4.08)},
  {'text': ' real', 'timestamp': (4.08, 4.4)},
  {'text': ' life.', 'timestamp': (4.4, 5.28)},
  {'text': ' Grab', 'timestamp': (5.64, 5.64)},
  {'text': ' yours', 'timestamp': (5.64, 6.04)},
  {'text': ' at', 'timestamp': (6.04, 6.36)},
  {'text': ' Target,', 'timestamp': (6.36, 7.16)},
  {'text': ' Costco,', 'timestamp': (7.16, 8.08)},
  {'text': ' or', 'timestamp': (8.08, 8.3)},
  {'text': ' head', 'timestamp': (8.3, 8.5)},
  {'text': ' to', 'timestamp': (8.5, 8.7)},
  {'text': ' TrueClassic', 'timestamp': (8.7, 9.32)},
  {'text': '.com', 'timestamp': (9.32, 10.02)},
  {'text': ' slash', 'timestamp': (10.02, 10.78)},
  {'text': ' P4P.', 'timestamp': (10.78, 12.74)},
  {'text': ' Get', 'timestamp': (12.74, 13.06)},
  {'text': ' hooked', 'timestamp': (13.06, 13.3)},
  {'text': ' up', 'timestamp': (13.3, 13.52)},
  {'text': ' today.', 'timestamp': (13.52, 14.16)},
  {'text': ' Now', 'timestamp': (14.18, 14.24)},
  {'text': ' before', 'timestamp': (14.24, 14.44)},
  {'text': ' we', 'timestamp': (14.44, 14.62)},
  {'text': ' go,', 'timestamp': (14.62, 15.08)},
  {'text': ' just', 'timestamp': (15.08, 15.24)},
  {'text': ' want', 'timestamp': (15.24, 15.42)},
  {'text': ' to', 'timestamp': (15.42, 15.46)},
  {'text': ' give', 'timestamp': (15.46, 15.56)},
  {'text': ' a', 'timestamp': (15.56, 15.68)},
  {'text': ' big', 'timestamp': (15.68, 15.86)},
  {'text': ' shout', 'timestamp': (15.86, 16.1)},
  {'text': ' out', 'timestamp': (16.1, 16.28)},
  {'text': ' to', 'timestamp': (16.28, 16.74)},
  {'text': ' the', 'timestamp': (16.74, 16.9)},
  {'text': ' CEO', 'timestamp': (16.9, 17.52)},
  {'text': ' and', 'timestamp': (17.52, 17.86)},
  {'text': ' founder', 'timestamp': (17.86, 18.3)},
  {'text': ' Ryan', 'timestamp': (18.3, 18.6)},
  {'text': ' brother', 'timestamp': (18.72, 18.86)},
  {'text': ' for', 'timestamp': (18.86, 19.06)},
  {'text': ' coming', 'timestamp': (19.06, 19.2)},
  {'text': ' on', 'timestamp': (19.2, 19.36)},
  {'text': ' our', 'timestamp': (19.36, 19.5)},
  {'text': ' show', 'timestamp': (19.5, 19.74)},
  {'text': ' and', 'timestamp': (19.74, 20.4)},
  {'text': ' just', 'timestamp': (20.4, 20.6)},
  {'text': ' showing', 'timestamp': (20.6, 20.86)},
  {'text': ' some', 'timestamp': (20.86, 21.08)},
  {'text': ' love.', 'timestamp': (21.08, 21.46)},
  {'text': ' Now,', 'timestamp': (21.46, 21.86)},
  {'text': " let's", 'timestamp': (21.86, 22.36)},
  {'text': ' get', 'timestamp': (22.36, 22.46)},
  {'text': ' back', 'timestamp': (22.46, 22.68)},
  {'text': ' to', 'timestamp': (22.68, 23.06)},
  {'text': ' the', 'timestamp': (23.06, 23.2)},
  {'text': ' episode.', 'timestamp': (23.2, 24.24)},
  {'text': ' I', 'timestamp': (24.24, 24.44)},
  {'text': ' mean,', 'timestamp': (24.44, 25.06)},
  {'text': ' like', 'timestamp': (25.06, 25.42)},
  {'text': ' I', 'timestamp': (25.42, 25.52)},
  {'text': ' said,', 'timestamp': (25.52, 25.8)},
  {'text': " we're", 'timestamp': (25.8, 26.1)},
  {'text': ' going', 'timestamp': (26.1, 26.34)},
  {'text': ' through', 'timestamp': (26.34, 26.58)},
  {'text': ' that.', 'timestamp': (26.58, 27.08)},
  {'text': " We're", 'timestamp': (27.08, 27.34)},
  {'text': ' losing', 'timestamp': (27.34, 27.52)},
  {'text': ' stars.', 'timestamp': (27.52, 28.6)},
  {'text': ' And', 'timestamp': (28.16, 29.26)},
  {'text': ' then', 'timestamp': (29.26, 29.38)},
  {'text': ' we', 'timestamp': (29.38, 29.58)},
  {'text': ' kind', 'timestamp': (29.58, 29.84)},
  {'text': ' of', 'timestamp': (29.84, 29.98)}]}

[xenova]

Possibly related to huggingface/transformers#36632

[jozefchutka]

I am not sure it is related to python transformers, considering #1358 (comment) produces correct result while transformers.js does not.

[xenova]

(yes, I mean it was probably fixed in the python version with that update :) )

[jozefchutka]

Ah, understood. What is the process to get the python update into trasformers.js?

[jozefchutka]

I wanted to kindly bump this issue because it's affecting real-world use cases where Whisper models (base, small, and large) often return broken timestamps or skip large chunks of transcription. The same input transcribes correctly using the original PyTorch model in transformers, but something seems to go wrong with transformers.js (also in the most recent v3.7.0).

There are quite a few scenarios where even simple, clear speech causes large blank gaps or misaligned segments in the output. It feels like something may be off in the decoder logic or token alignment.

It would be amazing if someone from the team could take a look — I'm happy to help test or provide more examples if needed.

[jozefchutka]

I’ve spent some time digging into this and found a couple of differences between the Python and JavaScript implementations that seem to affect output quality:

One important fix is related to the PR huggingface/transformers#36632, which isn't reflected in the JS version yet.

Another critical piece seems to be missing spectrogram trimming (as done in Python before returning the final features).

In particular, this section from the Python version really helps improve alignment and output quality:
https://github.com/huggingface/transformers/blob/0d511f7a771c24acc2400a0b83c225eb16afe47d/src/transformers/models/whisper/feature_extraction_whisper.py#L131

Here’s the relevant part of that logic in Python, which could be adapted for JS:

log_spec = log_spec[:, :-1]

Adding a similar trimming step after computing log_spec in JS (e.g., removing the last time frame) brought my output much closer to the Python baseline.

Code to be added to line https://github.com/huggingface/transformers.js/blob/main/src/models/whisper/feature_extraction_whisper.js#L48

features = features.slice([0, features.dims[0]], [0, features.dims[1] - 1]);

EDIT 2025-08-05:

Spending a bit more time with this now (base model + word timestamps), it is far more reliable to just simply "mute" the first and the last bytes of waveform instead of features.slice() (my understanding now is that the trimming is actually done inside spectrum())

const _extract_fbank_features = WhisperFeatureExtractor.prototype._extract_fbank_features;
let extractedChunks = 0;
WhisperFeatureExtractor.prototype._extract_fbank_features = async function (waveform) {
	if(notFirstChunk)
		waveform.fill(0, 0, 16_000);
	if(notLastChunk)
		waveform.fill(0, waveform.length - 32_000);
	return _extract_fbank_features.call(this, waveform);
}

With this patch in place I can see word level timestamps are generated for following audio https://www.youtube.com/watch?v=EwTZ2xpQwpA

I am not sure why muting these parts improves the transcript, maybe it is spectrum calculation or token merging along strides.