Skip to content

[Feat] multimodal: [sglang-miles] Let a token-space strategy load its own media and choose its image decoder and RGB conversion - #48953

Merged
Rockdu merged 8 commits into
sglang-milesfrom
rockdu-dev-mm-v2-miles/17-image-load-options
Nov 8, 2026
Merged

Rockdu merged 8 commits into
sglang-milesfrom
rockdu-dev-mm-v2-miles/17-image-load-options

Conversation

@Rockdu

@Rockdu Rockdu commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

What

Token-space processor only. The legacy processors, the EPD receiver and the embedded Rust MM pipeline keep their current media loading.

  • The mixin's per-modality media loaders become module-level functions (load_image_item, load_video_item, load_audio_item, with media_load_errors mapping loader exceptions to client errors). The legacy mixin calls them with the options it had before (rgb_conversion="smart" if cls.smart_rgb_conversion else "convert"), so its behavior does not change.
  • A TokenSpaceProcessStrategy loads its own media through load_image, load_video and load_audio. The token-space processor routes every submit path through them, so a model's strategy can override how its media are loaded.
  • Two loader options live on the strategy and are resolved once, at construction, from two top-level --mm-process-config keys (next to video_preprocessing_device):
    • image_decoder: pil, nvjpeg or nvjpeg_fancy, the JPEG decoder (gpu_image_decode).
    • image_rgb_conversion: convert, white or smart, how an image with alpha becomes RGB. white composites onto a white background exactly as qwen_vl_utils does (only for mode RGBA); smart is the GLM heuristic the legacy processors call smart_rgb_conversion.
  • QwenTokenSpaceProcessStrategy sets rgb_conversion = "white": the Qwen-VL family was trained on qwen_vl_utils pixels, which composite alpha onto white; the legacy Qwen processor keeps convert("RGB").
  • The two keys are validated at argument resolution (serving_hook), rejected without --enable-token-space-processor, and rejected when the model is served by a legacy processor, so a key no code reads cannot be set silently.

Why

A training framework that replays the engine's preprocessing on the trainer side needs the same pixels on both sides. The engine decoded JPEGs with nvJPEG and dropped alpha with convert("RGB"), while qwen_vl_utils decodes with PIL and composites RGBA onto white; both were class attributes with no server-level override. Putting the loaders on the strategy makes the per-model choice explicit and overridable per deployment, without touching the legacy processors' code paths.

The keys are top-level because the image / video / audio sub-dicts are forwarded verbatim as the HuggingFace processor's images_kwargs / videos_kwargs / audio_kwargs; a loader option inside image would reach the HF image processor and fail its kwargs validation (transformers 5.x raises on unknown keys).

Behavior changes

  • Legacy processors: none. smart_rgb_conversion keeps its name and meaning; the extracted loaders receive the same arguments the mixin passed before.
  • Token-space Qwen-VL: an image with an alpha channel is composited onto white instead of convert("RGB"). RGB images (every JPEG) are unaffected; the geo3k recipes are not affected.
  • mm_process_config["image_decoder"] / ["image_rgb_conversion"] are new keys; setting them without the token-space processor, or for a model a legacy processor serves, is an error.
  • Two kwargs of the token-space _submit_mm_data_loading_tasks_simple (audio_sample_rate, discard_alpha_channel) only mirror the legacy mixin's signature; the strategy holds both. Marked with a TODO to drop with that signature.

Validation

CPU unit tests (no model, no server), both registered in base-a-test-cpu:

pytest test/registered/unit/multimodal/test_image_load_options.py \
       test/registered/unit/multimodal/test_mm_token_expansion.py

test_image_load_options.py (new): the resolvers accept the three decoders and three conversions and reject anything else, white composites only an alpha channel (RGBA on white; LA and palette PNGs with a tRNS entry go through convert("RGB"), as in qwen_vl_utils), the loader options reach the IO path, and a corrupt image surfaces as a client error (UnidentifiedImageError). test_mm_token_expansion.py: TestStrategyLoaders pins the strategy loader contract (per-strategy defaults, Qwen white, config override, real RGBA file through the token-space path), defined before the __main__ block so the CI runner executes it.

End to end (radixark/miles, colocated RL): an Inkling-Small LoRA audio run, 50 rollouts, with this branch plus the Inkling token-space strategy as the engine; train/rollout logprob abs diff 0.013–0.015 throughout, matching the reference run on the previous engine, held-out AVQA 55.4 → 71.7 (reference 55.1 → 72.8), MMAU 78.3 → 77.3 (reference 78.9 → 76.9).

Landing

Base is sglang-miles at f8d0201. The same change for main is #48954. The Inkling token-space strategy builds on this branch and follows in its own PR.


CI States

Latest PR Test (Base): ❌ Run #42422890264
Latest PR Test (Extra): ❌ Run #42422890237
Latest PR Test (AMD ROCm 10): ❌ Run #42422890258

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@github-actions github-actions Bot added documentation Improvements or additions to documentation Multi-modal multi-modal language model labels Nov 6, 2026
@Rockdu Rockdu changed the title [Feature] multimodal: [sglang-miles] make the image decoder and RGB conversion mm_process_config options and composite Qwen-VL alpha onto white [Feat] multimodal: [sglang-miles] Make the image decoder and RGB conversion mm_process_config options and composite Qwen-VL alpha onto white Nov 6, 2026
@Rockdu
Rockdu force-pushed the rockdu-dev-mm-v2-miles/17-image-load-options branch from 4a4318e to a73b3c4 Compare November 6, 2026 19:23
@Rockdu Rockdu changed the title [Feat] multimodal: [sglang-miles] Make the image decoder and RGB conversion mm_process_config options and composite Qwen-VL alpha onto white [Fix] qwen_vl: [sglang-miles] Composite alpha onto white as qwen_vl_utils does, with the image decoder and RGB conversion as mm_process_config options Nov 6, 2026
@Rockdu
Rockdu force-pushed the rockdu-dev-mm-v2-miles/17-image-load-options branch from a73b3c4 to 020924b Compare November 6, 2026 19:31
@Rockdu Rockdu changed the title [Fix] qwen_vl: [sglang-miles] Composite alpha onto white as qwen_vl_utils does, with the image decoder and RGB conversion as mm_process_config options [Feat] multimodal: [sglang-miles] Make the image decoder and RGB conversion mm_process_config options and fix Qwen-VL's RGBA handling Nov 6, 2026
@Rockdu
Rockdu force-pushed the rockdu-dev-mm-v2-miles/17-image-load-options branch from 020924b to 68413a3 Compare November 6, 2026 19:33
@Rockdu
Rockdu added this pull request to stack #48553 November 6, 2026 19:47
@Rockdu
Rockdu force-pushed the rockdu-dev-mm-v2-miles/17-image-load-options branch 4 times, most recently from b5f660f to 633dd1e Compare November 6, 2026 21:12
Base automatically changed from rockdu-dev-mm-v2-miles/16-item-feature-cache-write-once to sglang-miles November 6, 2026 21:12
@Rockdu
Rockdu force-pushed the rockdu-dev-mm-v2-miles/17-image-load-options branch from 633dd1e to d528a05 Compare November 6, 2026 21:12
…y media loaders

The three branches of `_load_single_item` and its error classification
become module functions, so other loaders can reuse them; behavior unchanged.

(cherry picked from commit 4e1cf4a2e3df3fc3e8f8f06272ed4b37260d079a)

Signed-off-by: rockdu <kangrdu@gmail.com>
… own media

The base strategy gains `load_image`, `load_video` and `load_audio`, whose
defaults are the shared loaders with the strategy's declared options; the
core submits every source to them instead of the mixin's loader, keeping the
mixin's error classification. A model whose processors decode the media
themselves overrides the hook for that modality. The hooks run on the IO
threads, which share the strategy instance, so they read only class
attributes and construction state.

(cherry picked from commit f8641b9ab33611c9bf3b1498d7d3fd3024166019)

Signed-off-by: rockdu <kangrdu@gmail.com>
…ts image decoder and RGB conversion, Qwen-VL compositing alpha onto white

A strategy declares `gpu_image_decode` and `rgb_conversion` (convert, white or smart) and resolves `mm_process_config["image_decoder"]` / `["image_rgb_conversion"]` over them at construction, so its `load_image` decodes as the model's reference pipeline does and the trainer that replays the engine's preprocessing gets the same pixels. `QwenTokenSpaceProcessStrategy` defaults to `white`, as `qwen_vl_utils` composites alpha. The shared `load_image_item` takes the conversion by name; legacy processors keep `smart_rgb_conversion` and their behavior. The two keys are rejected without `--enable-token-space-processor`, since no legacy loader reads them.

Signed-off-by: rockdu <kangrdu@gmail.com>
…r keys at argument time and reject them where no strategy reads them

`handle_multimodal` resolves `image_decoder` / `image_rgb_conversion` with the token-space processor on, so a wrong value fails at argument resolution instead of when the tokenizer manager builds the processor; both resolvers take JSON null as the default and reject a non-string with the same ValueError. `get_mm_processor_cls` raises when the flag is on but the model is served by a legacy processor, which never reads the keys.

Signed-off-by: rockdu <kangrdu@gmail.com>
…lient error classes, loader options on the IO path, Qwen's white default

The failure-classification test raises an undecodable-image error, which only `media_load_errors` turns into a ValueError; the loader-options test runs an RGBA PNG through `TokenSpaceProcessor.process` and checks what `process_image` receives for the strategy's `discard_alpha_channel`, the server's `image_rgb_conversion` and the default; the legacy-defaults parity test compares `rgb_conversion` by name and records Qwen's `white` as the one declared divergence, so reverting it fails.

Signed-off-by: rockdu <kangrdu@gmail.com>
…ge with isort and ruff

Signed-off-by: rockdu <kangrdu@gmail.com>
…he __main__ block so the CI runner executes it

The registered CI runner runs a test file as a script, and `unittest.main()` exits before a class defined below it is created; pytest collected the class, which is why a local pytest run was green.

Signed-off-by: rockdu <kangrdu@gmail.com>
…token-space submit keeps only for the legacy mixin signature

Signed-off-by: rockdu <kangrdu@gmail.com>
@Rockdu
Rockdu force-pushed the rockdu-dev-mm-v2-miles/17-image-load-options branch from d528a05 to 80994be Compare November 7, 2026 22:55
@Rockdu Rockdu added the run-ci CI: run the baseline test suite on this PR label Nov 7, 2026
@Rockdu Rockdu changed the title [Feat] multimodal: [sglang-miles] Make the image decoder and RGB conversion mm_process_config options and fix Qwen-VL's RGBA handling [Feat] multimodal: [sglang-miles] Let a token-space strategy load its own media and choose its image decoder and RGB conversion Nov 7, 2026
@Rockdu
Rockdu merged commit 3bc84e3 into sglang-miles Nov 8, 2026
56 of 64 checks passed
@Rockdu
Rockdu deleted the rockdu-dev-mm-v2-miles/17-image-load-options branch November 8, 2026 07:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation Multi-modal multi-modal language model run-ci CI: run the baseline test suite on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant