Repository navigation
[Feat] multimodal: [sglang-miles] Let a token-space strategy load its own media and choose its image decoder and RGB conversion - #48953
Merged
Rockdu merged 8 commits intoNov 8, 2026
Conversation
Rockdu
requested review from
HaiShaw,
JustinTong0323,
mickqian,
sogalin,
wisclmy0611,
yctseng0211,
yhyang201,
yuan-luo and
zijiexia
as code owners
November 6, 2026 19:22
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
Rockdu
force-pushed
the
rockdu-dev-mm-v2-miles/17-image-load-options
branch
from
November 6, 2026 19:23
4a4318e to
a73b3c4
Compare
Rockdu
force-pushed
the
rockdu-dev-mm-v2-miles/17-image-load-options
branch
from
November 6, 2026 19:31
a73b3c4 to
020924b
Compare
Rockdu
force-pushed
the
rockdu-dev-mm-v2-miles/17-image-load-options
branch
from
November 6, 2026 19:33
020924b to
68413a3
Compare
Rockdu
force-pushed
the
rockdu-dev-mm-v2-miles/17-image-load-options
branch
4 times, most recently
from
November 6, 2026 21:12
b5f660f to
633dd1e
Compare
Base automatically changed from
rockdu-dev-mm-v2-miles/16-item-feature-cache-write-once
to
sglang-miles
November 6, 2026 21:12
Rockdu
force-pushed
the
rockdu-dev-mm-v2-miles/17-image-load-options
branch
from
November 6, 2026 21:12
633dd1e to
d528a05
Compare
5 of 6 tasks
…y media loaders The three branches of `_load_single_item` and its error classification become module functions, so other loaders can reuse them; behavior unchanged. (cherry picked from commit 4e1cf4a2e3df3fc3e8f8f06272ed4b37260d079a) Signed-off-by: rockdu <kangrdu@gmail.com>
… own media The base strategy gains `load_image`, `load_video` and `load_audio`, whose defaults are the shared loaders with the strategy's declared options; the core submits every source to them instead of the mixin's loader, keeping the mixin's error classification. A model whose processors decode the media themselves overrides the hook for that modality. The hooks run on the IO threads, which share the strategy instance, so they read only class attributes and construction state. (cherry picked from commit f8641b9ab33611c9bf3b1498d7d3fd3024166019) Signed-off-by: rockdu <kangrdu@gmail.com>
…ts image decoder and RGB conversion, Qwen-VL compositing alpha onto white A strategy declares `gpu_image_decode` and `rgb_conversion` (convert, white or smart) and resolves `mm_process_config["image_decoder"]` / `["image_rgb_conversion"]` over them at construction, so its `load_image` decodes as the model's reference pipeline does and the trainer that replays the engine's preprocessing gets the same pixels. `QwenTokenSpaceProcessStrategy` defaults to `white`, as `qwen_vl_utils` composites alpha. The shared `load_image_item` takes the conversion by name; legacy processors keep `smart_rgb_conversion` and their behavior. The two keys are rejected without `--enable-token-space-processor`, since no legacy loader reads them. Signed-off-by: rockdu <kangrdu@gmail.com>
…r keys at argument time and reject them where no strategy reads them `handle_multimodal` resolves `image_decoder` / `image_rgb_conversion` with the token-space processor on, so a wrong value fails at argument resolution instead of when the tokenizer manager builds the processor; both resolvers take JSON null as the default and reject a non-string with the same ValueError. `get_mm_processor_cls` raises when the flag is on but the model is served by a legacy processor, which never reads the keys. Signed-off-by: rockdu <kangrdu@gmail.com>
…lient error classes, loader options on the IO path, Qwen's white default The failure-classification test raises an undecodable-image error, which only `media_load_errors` turns into a ValueError; the loader-options test runs an RGBA PNG through `TokenSpaceProcessor.process` and checks what `process_image` receives for the strategy's `discard_alpha_channel`, the server's `image_rgb_conversion` and the default; the legacy-defaults parity test compares `rgb_conversion` by name and records Qwen's `white` as the one declared divergence, so reverting it fails. Signed-off-by: rockdu <kangrdu@gmail.com>
…ge with isort and ruff Signed-off-by: rockdu <kangrdu@gmail.com>
…he __main__ block so the CI runner executes it The registered CI runner runs a test file as a script, and `unittest.main()` exits before a class defined below it is created; pytest collected the class, which is why a local pytest run was green. Signed-off-by: rockdu <kangrdu@gmail.com>
…token-space submit keeps only for the legacy mixin signature Signed-off-by: rockdu <kangrdu@gmail.com>
Rockdu
force-pushed
the
rockdu-dev-mm-v2-miles/17-image-load-options
branch
from
November 7, 2026 22:55
d528a05 to
80994be
Compare
Rockdu
requested review from
Ying1123,
hnyls2002,
merrymercy and
xiezhq-hermann
as code owners
November 7, 2026 22:55
This was referenced Nov 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Token-space processor only. The legacy processors, the EPD receiver and the embedded Rust MM pipeline keep their current media loading.
load_image_item,load_video_item,load_audio_item, withmedia_load_errorsmapping loader exceptions to client errors). The legacy mixin calls them with the options it had before (rgb_conversion="smart" if cls.smart_rgb_conversion else "convert"), so its behavior does not change.TokenSpaceProcessStrategyloads its own media throughload_image,load_videoandload_audio. The token-space processor routes every submit path through them, so a model's strategy can override how its media are loaded.--mm-process-configkeys (next tovideo_preprocessing_device):image_decoder:pil,nvjpegornvjpeg_fancy, the JPEG decoder (gpu_image_decode).image_rgb_conversion:convert,whiteorsmart, how an image with alpha becomes RGB.whitecomposites onto a white background exactly asqwen_vl_utilsdoes (only for modeRGBA);smartis the GLM heuristic the legacy processors callsmart_rgb_conversion.QwenTokenSpaceProcessStrategysetsrgb_conversion = "white": the Qwen-VL family was trained onqwen_vl_utilspixels, which composite alpha onto white; the legacy Qwen processor keepsconvert("RGB").serving_hook), rejected without--enable-token-space-processor, and rejected when the model is served by a legacy processor, so a key no code reads cannot be set silently.Why
A training framework that replays the engine's preprocessing on the trainer side needs the same pixels on both sides. The engine decoded JPEGs with nvJPEG and dropped alpha with
convert("RGB"), whileqwen_vl_utilsdecodes with PIL and composites RGBA onto white; both were class attributes with no server-level override. Putting the loaders on the strategy makes the per-model choice explicit and overridable per deployment, without touching the legacy processors' code paths.The keys are top-level because the
image/video/audiosub-dicts are forwarded verbatim as the HuggingFace processor'simages_kwargs/videos_kwargs/audio_kwargs; a loader option insideimagewould reach the HF image processor and fail its kwargs validation (transformers 5.x raises on unknown keys).Behavior changes
smart_rgb_conversionkeeps its name and meaning; the extracted loaders receive the same arguments the mixin passed before.convert("RGB"). RGB images (every JPEG) are unaffected; the geo3k recipes are not affected.mm_process_config["image_decoder"]/["image_rgb_conversion"]are new keys; setting them without the token-space processor, or for a model a legacy processor serves, is an error._submit_mm_data_loading_tasks_simple(audio_sample_rate,discard_alpha_channel) only mirror the legacy mixin's signature; the strategy holds both. Marked with a TODO to drop with that signature.Validation
CPU unit tests (no model, no server), both registered in
base-a-test-cpu:test_image_load_options.py(new): the resolvers accept the three decoders and three conversions and reject anything else,whitecomposites only an alpha channel (RGBA on white;LAand palette PNGs with atRNSentry go throughconvert("RGB"), as inqwen_vl_utils), the loader options reach the IO path, and a corrupt image surfaces as a client error (UnidentifiedImageError).test_mm_token_expansion.py:TestStrategyLoaderspins the strategy loader contract (per-strategy defaults, Qwenwhite, config override, real RGBA file through the token-space path), defined before the__main__block so the CI runner executes it.End to end (radixark/miles, colocated RL): an Inkling-Small LoRA audio run, 50 rollouts, with this branch plus the Inkling token-space strategy as the engine; train/rollout logprob abs diff 0.013–0.015 throughout, matching the reference run on the previous engine, held-out AVQA 55.4 → 71.7 (reference 55.1 → 72.8), MMAU 78.3 → 77.3 (reference 78.9 → 76.9).
Landing
Base is
sglang-milesat f8d0201. The same change formainis #48954. The Inkling token-space strategy builds on this branch and follows in its own PR.CI States
Latest PR Test (Base): ❌ Run #42422890264
Latest PR Test (Extra): ❌ Run #42422890237
Latest PR Test (AMD ROCm 10): ❌ Run #42422890258
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: