Skip to content

[Feat] multimodal: Let a token-space strategy load its own media and choose its image decoder and RGB conversion - #48954

Open
Rockdu wants to merge 8 commits into
rockdu-dev-mm-v2/15-item-feature-cache-write-oncefrom
rockdu-dev-mm-v2/16-image-load-options
Open

Rockdu wants to merge 8 commits into
rockdu-dev-mm-v2/15-item-feature-cache-write-oncefrom
rockdu-dev-mm-v2/16-image-load-options

Conversation

@Rockdu

@Rockdu Rockdu commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

What

Token-space processor only. The legacy processors, the EPD receiver and the embedded Rust MM pipeline keep their current media loading.

  • The mixin's per-modality media loaders become module-level functions (load_image_item, load_video_item, load_audio_item, with media_load_errors mapping loader exceptions to client errors). The legacy mixin calls them with the options it had before (rgb_conversion="smart" if cls.smart_rgb_conversion else "convert"), so its behavior does not change.
  • A TokenSpaceProcessStrategy loads its own media through load_image, load_video and load_audio. The token-space processor routes every submit path through them, so a model's strategy can override how its media are loaded.
  • Two loader options live on the strategy and are resolved once, at construction, from two top-level --mm-process-config keys (next to video_preprocessing_device):
    • image_decoder: pil, nvjpeg or nvjpeg_fancy, the JPEG decoder (gpu_image_decode).
    • image_rgb_conversion: convert, white or smart, how an image with alpha becomes RGB. white composites onto a white background exactly as qwen_vl_utils does (only for mode RGBA); smart is the GLM heuristic the legacy processors call smart_rgb_conversion.
  • QwenTokenSpaceProcessStrategy sets rgb_conversion = "white": the Qwen-VL family was trained on qwen_vl_utils pixels, which composite alpha onto white; the legacy Qwen processor keeps convert("RGB").
  • The two keys are validated at argument resolution (serving_hook), rejected without --enable-token-space-processor, and rejected when the model is served by a legacy processor, so a key no code reads cannot be set silently.

Why

A training framework that replays the engine's preprocessing on the trainer side needs the same pixels on both sides. The engine decoded JPEGs with nvJPEG and dropped alpha with convert("RGB"), while qwen_vl_utils decodes with PIL and composites RGBA onto white; both were class attributes with no server-level override. Putting the loaders on the strategy makes the per-model choice explicit and overridable per deployment, without touching the legacy processors' code paths.

The keys are top-level because the image / video / audio sub-dicts are forwarded verbatim as the HuggingFace processor's images_kwargs / videos_kwargs / audio_kwargs; a loader option inside image would reach the HF image processor and fail its kwargs validation (transformers 5.x raises on unknown keys).

Behavior changes

  • Legacy processors: none. smart_rgb_conversion keeps its name and meaning; the extracted loaders receive the same arguments the mixin passed before.
  • Token-space Qwen-VL: an image with an alpha channel is composited onto white instead of convert("RGB"). RGB images (every JPEG) are unaffected; the geo3k recipes are not affected.
  • mm_process_config["image_decoder"] / ["image_rgb_conversion"] are new keys; setting them without the token-space processor, or for a model a legacy processor serves, is an error.
  • Two kwargs of the token-space _submit_mm_data_loading_tasks_simple (audio_sample_rate, discard_alpha_channel) only mirror the legacy mixin's signature; the strategy holds both. Marked with a TODO to drop with that signature.

Validation

CPU unit tests (no model, no server), both registered in base-a-test-cpu:

pytest test/registered/unit/multimodal/test_image_load_options.py \
       test/registered/unit/multimodal/test_mm_token_expansion.py

test_image_load_options.py (new): the resolvers accept the three decoders and three conversions and reject anything else, white composites only an alpha channel (RGBA on white; LA and palette PNGs with a tRNS entry go through convert("RGB"), as in qwen_vl_utils), the loader options reach the IO path, and a corrupt image surfaces as a client error (UnidentifiedImageError). test_mm_token_expansion.py: TestStrategyLoaders pins the strategy loader contract (per-strategy defaults, Qwen white, config override, real RGBA file through the token-space path), defined before the __main__ block so the CI runner executes it.

End to end (radixark/miles, colocated RL): an Inkling-Small LoRA audio run, 50 rollouts, with the sglang-miles twin of this branch (#48953, same change on sglang-miles f8d0201) plus the Inkling token-space strategy as the engine; train/rollout logprob abs diff 0.013–0.015 throughout, matching the reference run on the previous engine, held-out AVQA 55.4 → 71.7 (reference 55.1 → 72.8), MMAU 78.3 → 77.3 (reference 78.9 → 76.9).

Landing

Stacked on #15 of the token-space series (rockdu-dev-mm-v2/15-item-feature-cache-write-once). The same change for sglang-miles is #48953; the eight commits are identical apart from the base. The Inkling token-space strategy builds on this change and follows in its own PR.


CI States

Latest PR Test (Base): ❌ Run #42423323347
Latest PR Test (Extra): ❌ Run #42423323263
Latest PR Test (AMD ROCm 10): ❌ Run #42423323399

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@github-actions github-actions Bot added documentation Improvements or additions to documentation Multi-modal multi-modal language model labels Nov 6, 2026
@Rockdu Rockdu changed the title [Feature] multimodal: make the image decoder and RGB conversion mm_process_config options and composite Qwen-VL alpha onto white [Feat] multimodal: Make the image decoder and RGB conversion mm_process_config options and composite Qwen-VL alpha onto white Nov 6, 2026
@Rockdu
Rockdu force-pushed the rockdu-dev-mm-v2/16-image-load-options branch from c44ee3c to 8e07066 Compare November 6, 2026 19:23
@Rockdu Rockdu changed the title [Feat] multimodal: Make the image decoder and RGB conversion mm_process_config options and composite Qwen-VL alpha onto white [Fix] qwen_vl: Composite alpha onto white as qwen_vl_utils does, with the image decoder and RGB conversion as mm_process_config options Nov 6, 2026
@Rockdu
Rockdu force-pushed the rockdu-dev-mm-v2/16-image-load-options branch from 8e07066 to d3ac1bc Compare November 6, 2026 19:31
@Rockdu Rockdu changed the title [Fix] qwen_vl: Composite alpha onto white as qwen_vl_utils does, with the image decoder and RGB conversion as mm_process_config options [Feat] multimodal: Make the image decoder and RGB conversion mm_process_config options and fix Qwen-VL's RGBA handling Nov 6, 2026
@Rockdu
Rockdu force-pushed the rockdu-dev-mm-v2/16-image-load-options branch from d3ac1bc to 159d6de Compare November 6, 2026 19:33
The three branches of `_load_single_item` and its error classification
become module functions, so other loaders can reuse them; behavior unchanged.

Signed-off-by: rockdu <kangrdu@gmail.com>
The base strategy gains `load_image`, `load_video` and `load_audio`, whose
defaults are the shared loaders with the strategy's declared options; the
core submits every source to them instead of the mixin's loader, keeping the
mixin's error classification. A model whose processors decode the media
themselves overrides the hook for that modality. The hooks run on the IO
threads, which share the strategy instance, so they read only class
attributes and construction state.

Signed-off-by: rockdu <kangrdu@gmail.com>
…r and RGB conversion, Qwen-VL compositing alpha onto white

A strategy declares `gpu_image_decode` and `rgb_conversion` (convert, white or smart) and resolves `mm_process_config["image_decoder"]` / `["image_rgb_conversion"]` over them at construction, so its `load_image` decodes as the model's reference pipeline does and the trainer that replays the engine's preprocessing gets the same pixels. `QwenTokenSpaceProcessStrategy` defaults to `white`, as `qwen_vl_utils` composites alpha. The shared `load_image_item` takes the conversion by name; legacy processors keep `smart_rgb_conversion` and their behavior. The two keys are rejected without `--enable-token-space-processor`, since no legacy loader reads them.

Signed-off-by: rockdu <kangrdu@gmail.com>
…ent time and reject them where no strategy reads them

`handle_multimodal` resolves `image_decoder` / `image_rgb_conversion` with the token-space processor on, so a wrong value fails at argument resolution instead of when the tokenizer manager builds the processor; both resolvers take JSON null as the default and reject a non-string with the same ValueError. `get_mm_processor_cls` raises when the flag is on but the model is served by a legacy processor, which never reads the keys.

Signed-off-by: rockdu <kangrdu@gmail.com>
…sses, loader options on the IO path, Qwen's white default

The failure-classification test raises an undecodable-image error, which only `media_load_errors` turns into a ValueError; the loader-options test runs an RGBA PNG through `TokenSpaceProcessor.process` and checks what `process_image` receives for the strategy's `discard_alpha_channel`, the server's `image_rgb_conversion` and the default; the legacy-defaults parity test compares `rgb_conversion` by name and records Qwen's `white` as the one declared divergence, so reverting it fails.

Signed-off-by: rockdu <kangrdu@gmail.com>
…nd ruff

Signed-off-by: rockdu <kangrdu@gmail.com>
…ck so the CI runner executes it

The registered CI runner runs a test file as a script, and `unittest.main()` exits before a class defined below it is created; pytest collected the class, which is why a local pytest run was green.

Signed-off-by: rockdu <kangrdu@gmail.com>
…mit keeps only for the legacy mixin signature

Signed-off-by: rockdu <kangrdu@gmail.com>
@Rockdu
Rockdu force-pushed the rockdu-dev-mm-v2/16-image-load-options branch from 159d6de to ee95628 Compare November 7, 2026 23:01
@Rockdu Rockdu added the run-ci CI: run the baseline test suite on this PR label Nov 7, 2026
@Rockdu Rockdu changed the title [Feat] multimodal: Make the image decoder and RGB conversion mm_process_config options and fix Qwen-VL's RGBA handling [Feat] multimodal: Let a token-space strategy load its own media and choose its image decoder and RGB conversion Nov 7, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation Multi-modal multi-modal language model run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant