[Feature]: Eliminate redundant multimodal preprocessing for gRPC backends

[CatherineSue]

Problem Statement

When smg routes multimodal (VLM) requests via gRPC, the three backend engines handle image preprocessing very differently. Only SGLang and vLLM receives the full benefit of smg's Rust-based preprocessing — TensorRT-LLM still has some gaps to be filled later.

What smg's multimodal crate does today

smg's Rust multimodal crate handles the full image preprocessing pipeline:

1. Image loading — fetch/decode raw images
2. Image preprocessing — resize, normalize, crop (model-specific: Qwen, Qwen3, LLaVA, Phi3, Pixtral, LLaMA4, etc.)
3. Token expansion — compute how many tokens each image occupies (e.g., N = prod(image_grid_thw) // merge_size² for Qwen)
4. Model-specific metadata — produce auxiliary tensors (image_grid_thw for Qwen, aspect_ratios for LLaMA4, image_sizes for Phi3/LLaVA, etc.)

Output: pixel_values (preprocessed image tensor) + model_specific_tensors + expanded_token_ids + mm_placeholders

Current gRPC routing by backend

Updated — PR #588 replaced the unified MultimodalData struct with backend-specific variants. Assembly is now deferred to request_building after worker selection, so each backend only serializes the fields it needs.

process_multimodal() in multimodal.rs:
  fetch → preprocess → expand tokens → MultimodalIntermediate (lightweight, no serialization)
                                              ↓
                                    (worker selection determines backend)
                                              ↓
  assemble_multimodal_data(intermediate, client) →
    MultimodalData::Sglang  → pixel_values + model_specific + patch-only placeholders
    MultimodalData::Vllm    → pixel_values + model_specific + structural placeholders + hashes + field keys
    MultimodalData::Trtllm  → raw image bytes only (no tensor serialization)

What smg sends	SGLang	vLLM	TensorRT-LLM
Token IDs (expanded)	✅	✅	❌ (unexpanded)
pixel_values (tensor)	✅	✅	❌
model_specific_tensors	✅	✅	❌
mm_placeholders	✅ (patch-only)	✅ (structural)	❌
mm_hashes (blake3)	❌	✅	❌
batched_keys / flat_keys	❌	✅	❌
Raw image bytes	✅	❌	✅

---

How each engine processes multimodal data it receives

SGLang (most efficient — zero redundant work)

smg sends: expanded_token_ids + pixel_values + model_specific_tensors + mm_placeholders
    ↓
grpc_server._parse_mm_inputs()
    → Decodes TensorData proto into torch tensors
    → Creates MultimodalDataItem(feature=pixel_values, model_specific_data={image_grid_thw: ...})
    ↓
No HF processor called
    ↓
model.get_image_feature(items)
    → Uses item.feature (pixel_values) directly
    → Vision encoder (ViT) runs on GPU

Waste: None. smg does all preprocessing in Rust. SGLang just unwraps the tensors and feeds them to the ViT.

---

vLLM (solved by #570 — zero redundant work)

Before #570: vLLM received raw image bytes and re-ran HF processor (dummy tokenization + image resize/normalize/crop). See collapsed section below for the old flow.

smg sends: expanded_token_ids + pixel_values + model_specific_tensors
         + mm_placeholders + mm_hashes + batched_keys/flat_keys
    ↓
grpc_server._build_preprocessed_mm_inputs()
    → Decodes TensorData proto into torch tensors
    → Casts floating-point tensors to model dtype (bfloat16)
    → Constructs MultiModalFieldConfig from batched_keys/flat_keys
    → Builds is_embed mask on PlaceholderRange using im_token_id
    → Constructs MultiModalInputs via mm_inputs()
    ↓
No HF processor called
No dummy tokenization
    ↓
Vision encoder (ViT) runs on GPU

Waste: None. smg does all preprocessing in Rust. vLLM constructs MultiModalKwargsItems.from_hf_inputs(BatchFeature(hf_dict), fields_config) directly from the proto tensors.

Additional capabilities enabled:

• Encoder output caching — blake3 per-image hashes (mm_hashes) enable vLLM to cache and reuse ViT encoder outputs across requests with the same images
• Prefix caching — hash-based cache salt improves prefix cache hit rates for multimodal requests

Old vLLM flow (before #570)

smg sends: token_ids + raw image bytes
    ↓
grpc_server builds: TokensPrompt(prompt_token_ids=[...], multi_modal_data={"image": [PIL.Image]})
    ↓
Renderer.process_for_engine()
  → _process_tokens()
    → _process_multimodal(prompt_token_ids=[1,2,3], mm_data={"image": PIL.Image})
      → BaseMultiModalProcessor.apply(prompt=[1,2,3], mm_items=...)
        → _apply_hf_processor_main()
          ├─ _apply_hf_processor_tokens_only([1,2,3])  → returns [1,2,3] UNCHANGED
          └─ _apply_hf_processor_mm_only(images)
               → generates DUMMY TEXT                          ← wasted work
               → calls HF processor(text=dummy, images=real_images)  ← re-preprocesses images
               → discards tokenized output, keeps only pixel_values
    ↓
Vision encoder (ViT) runs on GPU

Waste:

• Image preprocessing repeated — smg could have sent pixel_values, but instead sends raw bytes → vLLM re-runs resize/normalize/crop via HF ImageProcessor
• Dummy tokenization — HF AutoProcessor requires text input, so vLLM generates dummy text, tokenizes it, then throws it away

---

TensorRT-LLM (worst — full double tokenization, not yet addressed)

smg sends: token_ids + raw image bytes
    ↓
grpc_servicer extracts images from proto
    ↓
llm.generate_async(inputs={"prompt_token_ids": [...], "multi_modal_data": {"image": [bytes]}})
    ↓
llm._preprocess()
    → Detects: token_ids + multi_modal_data + VLM model
    → prompt = tokenizer.decode(token_ids)         ← DECODES tokens back to text!
    → inputs = TextPrompt(prompt=text, multi_modal_data=images)
    ↓
VLM InputProcessor.__call__(text_prompt, images)
    → HF AutoProcessor(text=prompt, images=images)
      → image_processor(images) → pixel_values     ← re-preprocesses images
      → tokenizer(expanded_text) → input_ids        ← RE-TOKENIZES the prompt!
    → _postprocess(input_ids)
    → compute mrope, build multimodal_data
    ↓
Vision encoder (ViT) runs on GPU

Waste:

• Token IDs decoded back to text — tokenizer.decode(token_ids) undoes smg's tokenization
• Full re-tokenization — HF AutoProcessor re-tokenizes the decoded text (with image placeholder expansion)
• Image preprocessing repeated — HF ImageProcessor re-runs resize/normalize/crop on raw bytes
• All three steps that smg already did (tokenize, expand, image_process) are repeated

---

Solution

Phase 1: vLLM — Done (#570)

PR #570 sends preprocessed pixel_values + model_specific_tensors to vLLM via gRPC, matching the SGLang path. The two-phase pipeline split from #497 has been collapsed back to a single process_multimodal() entry point.

Phase 1.5: Backend-specific multimodal data — Done (#588)

PR #588 replaced the unified MultimodalData struct with an enum of backend-specific variants (SglangMultimodalData, VllmMultimodalData, TrtllmMultimodalData). Assembly is deferred to request_building after worker selection via assemble_multimodal_data(intermediate, client), so each backend only serializes the fields it needs. TRT-LLM no longer wastes time serializing megabytes of pixel_values it never sends.

Phase 2: TensorRT-LLM — Deferred

TRT-LLM proto expansion was prototyped and reverted in #570 — TRT-LLM backend-side changes to accept preprocessed tensors are not yet ready. The proto changes for TRT-LLM will be re-added when the backend support is implemented.

---

Related PRs

• feat(grpc): integrate multimodal processing into gRPC sglang chat pipeline #495 — feat(grpc): integrate multimodal processing into gRPC SGLang chat pipeline
• feat(grpc): add vLLM multimodal support and split pipeline into fetch + preprocess #497 — feat(grpc): add vLLM multimodal support and split pipeline into fetch + preprocess
• feat(gprc): add TensorRT-LLM multimodal support #504 — feat(grpc): add TensorRT-LLM multimodal support
• feat(grpc): send preprocessed multimodal data to vLLM with hashing and structured tokens #570 — feat(grpc): send preprocessed multimodal data to vLLM with hashing and structured tokens
• refactor(grpc): split MultimodalData into backend-specific variants #588 — refactor(grpc): split MultimodalData into backend-specific variants

Downstream PRs:

• SGLang: feat(grpc): add multimodal TensorData parsing for vision inference sgl-project/sglang#19535
• TensorRT-LLM: [#11578][feat] support multimodal image input in gRPC server NVIDIA/TensorRT-LLM#11800

Additional Context

Key code references

smg:

• process_multimodal() — async entry point: fetch → preprocess → expand tokens → MultimodalIntermediate
• assemble_multimodal_data() — deferred assembly: MultimodalIntermediate → backend-specific MultimodalData variant
• SglangMultimodalData::into_proto() — sends pixel_values + model_specific_tensors + patch-only placeholders
• VllmMultimodalData::into_proto() — sends pixel_values + model_specific_tensors + mm_hashes + batched_keys/flat_keys
• TrtllmMultimodalData::into_proto() — sends raw image bytes only
• multimodal/src/vision/processors/ — per-model image preprocessors (8 families)
• multimodal/src/registry.rs — token expansion specs (ModelProcessorSpec trait)
• multimodal/src/hasher.rs — blake3 image hashing

SGLang (feat/grpc-mm branch):

• _parse_mm_inputs() — decodes pixel_values from proto TensorData
• _decode_tensor_data() — TensorData proto → torch tensor

vLLM:

• _build_preprocessed_mm_inputs() — deserializes proto tensors, constructs MultiModalFieldConfig, builds is_embed mask
• MultiModalKwargsItems.from_hf_inputs() — constructs MM inputs from BatchFeature + field configs

TensorRT-LLM:

• tensorrt_llm/llmapi/llm.py:489-494 — the decode-and-reprocess path (tokenizer.decode() → HF AutoProcessor)
• tensorrt_llm/grpc/grpc_servicer.py — multimodal image extraction from proto

HF AutoProcessor internals (Qwen2VLProcessor as example)

The HF AutoProcessor.__call__() does 3 things:

1. self.image_processor(images) → pixel_values + image_grid_thw (CPU preprocessing: resize, normalize)
2. Text expansion — replace <|image_pad|> with N copies per image
3. self.tokenizer(expanded_text) → input_ids

It does NOT run the vision encoder (ViT). The ViT runs during model.forward().

FastImageProcessor

HuggingFace has FastImageProcessor (torchvision-based) that can run on GPU. SGLang uses it with device="cuda" by default. vLLM uses use_fast=True but defaults to CPU. This is orthogonal to the main issue — the savings come from skipping redundant preprocessing entirely, not from where resize/normalize runs.

[CatherineSue]

Related PRs:

• SGLang: feat(grpc): add multimodal TensorData parsing for vision inference sgl-project/sglang#19535
• TensorRT-LLM: [#11578][feat] support multimodal image input in gRPC server NVIDIA/TensorRT-LLM#11800

[CatherineSue]

Updated the issue description to reflect current state. Phase 1 for vLLM is implemented in #570 — smg now sends preprocessed data to vLLM, matching the SGLang path. TRT-LLM remains deferred.