Repository navigation
Description
Versions
- Dynamo 7e8b9fc (TRT-LLM backend, aggregated worker). The call path below is unchanged on
mainat 2c8c330 (2026-10-18). - TensorRT-LLM 1.3.0rc25 (newer releases not checked).
- Qwen3-VL-8B-Instruct; 10 s 720p video clips (about 12K visual tokens per request).
Summary
When a video request arrives, every request that is already decoding stops receiving tokens for 0.6–0.9 s on our hosts. On fast GPUs this is not GPU time: the engine keeps decoding through the gap, and the held-back tokens are delivered in a burst once the new request has been preprocessed. On those GPUs this host-side gap is the only stall left, and it occurs for every running stream each time a clip arrives.
Where it happens
components/src/dynamo/trtllm/request_handlers/handler_base.pycallsself.engine.llm.generate_async(...)from the async request handler, that is, on the worker's asyncio event loop. Onmainthe call is the same; the comment next to it already notes that TRT-LLM performs request validation and preprocessing synchronously insidegenerate_async().- In TensorRT-LLM 1.3.0rc25,
BaseLLM.generate_async()(tensorrt_llm/llmapi/llm.py) is a synchronous method. Before it submits the request to the executor, it callsself._preprocess(...), which runs the tokenizer and the HF multimodal processor (for Qwen3-VL video: resize, normalise, patchify). - Stand-alone, this preprocessing takes 0.36–0.74 s per clip on our hosts, depending on the CPU.
- In 7e8b9fc video decoding already runs off the loop (
asyncio.to_threadinasync_load_video); preprocessing does not. - While
_preprocessruns, no other coroutine runs, so theasync for res in generation_resultloops of all other requests cannot forward tokens.
Evidence
We timestamped every executor iteration on the host clock and aligned the stamps with the client-side token arrival times. Setup: aggregated worker, 8 concurrent streams, 48 requests, 4,096 output tokens, unchunked prefill.
| GPU | gaps > 0.5 s per request | where the gap time goes | median gap | engine work per admission |
|---|---|---|---|---|
| B200 | 5.1 | 247 / 247 gaps outside the engine | 0.69 s | 0.18 + 0.18 s |
| H200 | 4.0 | 192 / 192 gaps outside the engine | 0.89 s | 0.19 + 0.36 s |
| H100 | 4.0 | 191 / 191 gaps outside the engine | 0.90 s | 0.20 + 0.36 s |
What the iteration stamps show for each of these gaps:
- the engine runs a median 55–111 decode iterations and is busy 99% of the time;
- the new request is fetched at the very end of the gap (median relative position 1.00);
- the gap length tracks the host's preprocessing time, not the GPU.
Fix we tested
Call the public LLM.preprocess() in a worker thread, then pass the returned PreprocessedInputs to generate_async(), which skips preprocessing for that input type. This is also what trtllm-serve does for chat completions: it runs LLM.preprocess() on a thread pool before calling generate_async().
if isinstance(processed_input, dict) and processed_input.get("multi_modal_data"):
processed_input = await asyncio.to_thread(
self.engine.llm.preprocess, processed_input, sampling_params, disaggregated_params
)
generation_result = self.engine.llm.generate_async(inputs=processed_input, ...)Results: one control run and two patched runs per GPU, on the same node with the same request set (8 concurrent streams unless noted).
| GPU | gaps > 0.5 s per request | worst gap | throughput | TTFT p50 |
|---|---|---|---|---|
| B200, control | 5.00 | 2.40 s | — | 1.40 s |
| B200, patched (2 runs) | 0 / 0 | 0.41 / 0.41 s | −1.0% / −0.9% | 2.39 / 2.30 s |
| H100, control | 3.98 | 3.00 s | — | 2.47 s |
| H100, patched (2 runs) | 0.98 / 0.71 | 2.05 / 0.56 s | −1.9% / −0.7% | 2.72 / 2.38 s |
| H100, Poisson 0.10 req/s, control | 5.44 | 1.55 s | (open loop) | 2.28 s |
| H100, Poisson 0.10 req/s, patched (2 runs) | 0.08 / 0.10 | 0.56 / 0.56 s | (open loop) | 6.33 / 2.43 s * |
| Lower-compute Hopper GPU (141 GB HBM3e), control | 8.75 | 2.19 s | — | 4.92 s |
| Lower-compute Hopper GPU (141 GB HBM3e), patched (2 runs) | 8.79 / 7.90 | 2.18 / 2.19 s | −0.1% / −0.3% | 4.87 / 6.40 s |
* Near the KV-cache limit at this rate; TTFT varies by several seconds between identical runs.
- With Poisson arrivals (open loop, the same arrival schedule in all three runs), every arrival still froze the running streams for about 0.67 s in the control. With the change, 4–5 gaps of 0.52–0.56 s per run were left and none above 1 s.
- On the lower-compute Hopper GPU the change does not help. There the admission itself is about 2.2 s of engine work (encoder + prefill), so the streams stall for about the same total time (11.1 s per request without the change, 11.1 / 10.6 s with it). The fix matters where the engine is fast enough for the host-side gap to be the dominant one.
Open points for a proper fix
- Side effects of unblocking the loop.
- On H100, about 10% of admission iterations take ~0.54 s instead of 0.20 s, leaving gaps just above 0.5 s. These are back-to-back admissions of clips that arrive together: with the loop no longer serialising them, each admission iteration waits for the previous one's prefill. A 1 ms GIL switch interval did not change them; with Poisson arrivals they mostly disappear (0.08–0.10 gaps per request).
- Running the preprocessing on one dedicated thread (
ThreadPoolExecutor(max_workers=1)instead ofasyncio.to_thread) removes them in closed loop as well: 0.85 → 0.06 / 0 gaps > 0.5 s per request on H100, throughput −1.0% / −2.7%, TTFT p50 2.11 → 1.90 / 1.87 s. Clips that arrive together are then preprocessed in arrival order, so their admissions are spaced out. The last clip of a burst waits longer (preprocessing p90 3.1 s instead of 2.6 s, max 5.2 s instead of 2.8 s), so the number of workers is a trade-off. - In the thread, the arriving request's preprocessing takes 0.66–0.97 s (median per run, 14 patched runs) while the other streams keep receiving tokens. Under closed-loop bursts of 8 streams its p90 is 2.1–2.8 s (0.9 s with Poisson arrivals). The arriving request's TTFT p50 rose by 0.9–1.0 s on B200 and changed by −0.1 to +0.25 s on H100 (closed loop).
- A process pool, or preprocessing in the frontend or an encode worker, would keep this CPU work out of the serving process. Pacing admissions would address the back-to-back case.
- Request rejection. On
main,ValueError/TypeError/NotImplementedErrorraised synchronously bygenerate_async()are turned into a per-request error. With preprocessing in a thread, the same exceptions come from the awaited call and need the same handling. - Concurrent tokenizer use from two threads. We saw no errors in 672 patched requests (14 runs on three GPU types), but HF fast tokenizers can raise "Already borrowed" under concurrent use. The tokenizer's thread safety should be checked.
- Missing prompt text.
RequestOutput.promptisNoneon thePreprocessedInputspath; the handler does not read it today. - Other synchronous media steps on the loop. The same pattern probably affects image requests with large images, and any other synchronous work in the handler.
The open PR #15951 (2026-10-09) makes the same change as the fix above (asyncio.to_thread around LLM.preprocess()), motivated by blocked health checks; the measurements here are evidence for it from the streaming side. On top of it we would suggest a switch to turn the offload off, a dedicated single-thread executor for the back-to-back case (point 1), and the error handling in point 2. We are happy to test #15951 on our setup and to send a small follow-up PR for the dedicated executor if that is the preferred design.
Activity
john-liu-dev commentedon Nov 3, 2026
Related work: #17243 moves multimodal preprocessing off the event loop into a worker pool, sized by the
num_input_processor_workerssetting added in NVIDIA/TensorRT-LLM#20977.