Skip to content

[trtllm] Multimodal input preprocessing runs on the worker's event loop and freezes token streaming for every in-flight request #16768

Open

Description

@john-liu-dev

Versions

  • Dynamo 7e8b9fc (TRT-LLM backend, aggregated worker). The call path below is unchanged on main at 2c8c330 (2026-10-18).
  • TensorRT-LLM 1.3.0rc25 (newer releases not checked).
  • Qwen3-VL-8B-Instruct; 10 s 720p video clips (about 12K visual tokens per request).

Summary

When a video request arrives, every request that is already decoding stops receiving tokens for 0.6–0.9 s on our hosts. On fast GPUs this is not GPU time: the engine keeps decoding through the gap, and the held-back tokens are delivered in a burst once the new request has been preprocessed. On those GPUs this host-side gap is the only stall left, and it occurs for every running stream each time a clip arrives.

Where it happens

  • components/src/dynamo/trtllm/request_handlers/handler_base.py calls self.engine.llm.generate_async(...) from the async request handler, that is, on the worker's asyncio event loop. On main the call is the same; the comment next to it already notes that TRT-LLM performs request validation and preprocessing synchronously inside generate_async().
  • In TensorRT-LLM 1.3.0rc25, BaseLLM.generate_async() (tensorrt_llm/llmapi/llm.py) is a synchronous method. Before it submits the request to the executor, it calls self._preprocess(...), which runs the tokenizer and the HF multimodal processor (for Qwen3-VL video: resize, normalise, patchify).
  • Stand-alone, this preprocessing takes 0.36–0.74 s per clip on our hosts, depending on the CPU.
  • In 7e8b9fc video decoding already runs off the loop (asyncio.to_thread in async_load_video); preprocessing does not.
  • While _preprocess runs, no other coroutine runs, so the async for res in generation_result loops of all other requests cannot forward tokens.

Evidence

We timestamped every executor iteration on the host clock and aligned the stamps with the client-side token arrival times. Setup: aggregated worker, 8 concurrent streams, 48 requests, 4,096 output tokens, unchunked prefill.

GPU gaps > 0.5 s per request where the gap time goes median gap engine work per admission
B200 5.1 247 / 247 gaps outside the engine 0.69 s 0.18 + 0.18 s
H200 4.0 192 / 192 gaps outside the engine 0.89 s 0.19 + 0.36 s
H100 4.0 191 / 191 gaps outside the engine 0.90 s 0.20 + 0.36 s

What the iteration stamps show for each of these gaps:

  • the engine runs a median 55–111 decode iterations and is busy 99% of the time;
  • the new request is fetched at the very end of the gap (median relative position 1.00);
  • the gap length tracks the host's preprocessing time, not the GPU.

Fix we tested

Call the public LLM.preprocess() in a worker thread, then pass the returned PreprocessedInputs to generate_async(), which skips preprocessing for that input type. This is also what trtllm-serve does for chat completions: it runs LLM.preprocess() on a thread pool before calling generate_async().

if isinstance(processed_input, dict) and processed_input.get("multi_modal_data"):
    processed_input = await asyncio.to_thread(
        self.engine.llm.preprocess, processed_input, sampling_params, disaggregated_params
    )
generation_result = self.engine.llm.generate_async(inputs=processed_input, ...)

Results: one control run and two patched runs per GPU, on the same node with the same request set (8 concurrent streams unless noted).

GPU gaps > 0.5 s per request worst gap throughput TTFT p50
B200, control 5.00 2.40 s — 1.40 s
B200, patched (2 runs) 0 / 0 0.41 / 0.41 s −1.0% / −0.9% 2.39 / 2.30 s
H100, control 3.98 3.00 s — 2.47 s
H100, patched (2 runs) 0.98 / 0.71 2.05 / 0.56 s −1.9% / −0.7% 2.72 / 2.38 s
H100, Poisson 0.10 req/s, control 5.44 1.55 s (open loop) 2.28 s
H100, Poisson 0.10 req/s, patched (2 runs) 0.08 / 0.10 0.56 / 0.56 s (open loop) 6.33 / 2.43 s *
Lower-compute Hopper GPU (141 GB HBM3e), control 8.75 2.19 s — 4.92 s
Lower-compute Hopper GPU (141 GB HBM3e), patched (2 runs) 8.79 / 7.90 2.18 / 2.19 s −0.1% / −0.3% 4.87 / 6.40 s

* Near the KV-cache limit at this rate; TTFT varies by several seconds between identical runs.

  • With Poisson arrivals (open loop, the same arrival schedule in all three runs), every arrival still froze the running streams for about 0.67 s in the control. With the change, 4–5 gaps of 0.52–0.56 s per run were left and none above 1 s.
  • On the lower-compute Hopper GPU the change does not help. There the admission itself is about 2.2 s of engine work (encoder + prefill), so the streams stall for about the same total time (11.1 s per request without the change, 11.1 / 10.6 s with it). The fix matters where the engine is fast enough for the host-side gap to be the dominant one.

Open points for a proper fix

  1. Side effects of unblocking the loop.
    • On H100, about 10% of admission iterations take ~0.54 s instead of 0.20 s, leaving gaps just above 0.5 s. These are back-to-back admissions of clips that arrive together: with the loop no longer serialising them, each admission iteration waits for the previous one's prefill. A 1 ms GIL switch interval did not change them; with Poisson arrivals they mostly disappear (0.08–0.10 gaps per request).
    • Running the preprocessing on one dedicated thread (ThreadPoolExecutor(max_workers=1) instead of asyncio.to_thread) removes them in closed loop as well: 0.85 → 0.06 / 0 gaps > 0.5 s per request on H100, throughput −1.0% / −2.7%, TTFT p50 2.11 → 1.90 / 1.87 s. Clips that arrive together are then preprocessed in arrival order, so their admissions are spaced out. The last clip of a burst waits longer (preprocessing p90 3.1 s instead of 2.6 s, max 5.2 s instead of 2.8 s), so the number of workers is a trade-off.
    • In the thread, the arriving request's preprocessing takes 0.66–0.97 s (median per run, 14 patched runs) while the other streams keep receiving tokens. Under closed-loop bursts of 8 streams its p90 is 2.1–2.8 s (0.9 s with Poisson arrivals). The arriving request's TTFT p50 rose by 0.9–1.0 s on B200 and changed by −0.1 to +0.25 s on H100 (closed loop).
    • A process pool, or preprocessing in the frontend or an encode worker, would keep this CPU work out of the serving process. Pacing admissions would address the back-to-back case.
  2. Request rejection. On main, ValueError / TypeError / NotImplementedError raised synchronously by generate_async() are turned into a per-request error. With preprocessing in a thread, the same exceptions come from the awaited call and need the same handling.
  3. Concurrent tokenizer use from two threads. We saw no errors in 672 patched requests (14 runs on three GPU types), but HF fast tokenizers can raise "Already borrowed" under concurrent use. The tokenizer's thread safety should be checked.
  4. Missing prompt text. RequestOutput.prompt is None on the PreprocessedInputs path; the handler does not read it today.
  5. Other synchronous media steps on the loop. The same pattern probably affects image requests with large images, and any other synchronous work in the handler.

The open PR #15951 (2026-10-09) makes the same change as the fix above (asyncio.to_thread around LLM.preprocess()), motivated by blocked health checks; the measurements here are evidence for it from the streaming side. On top of it we would suggest a switch to turn the offload off, a dedicated single-thread executor for the back-to-back case (point 1), and the error handling in point 2. We are happy to test #15951 on our setup and to send a small follow-up PR for the dedicated executor if that is the preferred design.

Activity

  1. john-liu-dev commented on Nov 3, 2026

    @john-liu-dev
    ContributorAuthor

    Related work: #17243 moves multimodal preprocessing off the event loop into a worker pool, sized by the num_input_processor_workers setting added in NVIDIA/TensorRT-LLM#20977.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions