Repository navigation
[Feat] multimodal: Let a token-space strategy load its own media and choose its image decoder and RGB conversion - #48954
Open
Rockdu wants to merge 8 commits into
Conversation
Rockdu
requested review from
HaiShaw,
JustinTong0323,
mickqian,
sogalin,
wisclmy0611,
yctseng0211,
yhyang201,
yuan-luo and
zijiexia
as code owners
November 6, 2026 19:22
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.
Tip: disable this comment in your organization's Code Review settings.
Rockdu
force-pushed
the
rockdu-dev-mm-v2/16-image-load-options
branch
from
November 6, 2026 19:23
c44ee3c to
8e07066
Compare
Rockdu
force-pushed
the
rockdu-dev-mm-v2/16-image-load-options
branch
from
November 6, 2026 19:31
8e07066 to
d3ac1bc
Compare
Rockdu
force-pushed
the
rockdu-dev-mm-v2/16-image-load-options
branch
from
November 6, 2026 19:33
d3ac1bc to
159d6de
Compare
The three branches of `_load_single_item` and its error classification become module functions, so other loaders can reuse them; behavior unchanged. Signed-off-by: rockdu <kangrdu@gmail.com>
The base strategy gains `load_image`, `load_video` and `load_audio`, whose defaults are the shared loaders with the strategy's declared options; the core submits every source to them instead of the mixin's loader, keeping the mixin's error classification. A model whose processors decode the media themselves overrides the hook for that modality. The hooks run on the IO threads, which share the strategy instance, so they read only class attributes and construction state. Signed-off-by: rockdu <kangrdu@gmail.com>
…r and RGB conversion, Qwen-VL compositing alpha onto white A strategy declares `gpu_image_decode` and `rgb_conversion` (convert, white or smart) and resolves `mm_process_config["image_decoder"]` / `["image_rgb_conversion"]` over them at construction, so its `load_image` decodes as the model's reference pipeline does and the trainer that replays the engine's preprocessing gets the same pixels. `QwenTokenSpaceProcessStrategy` defaults to `white`, as `qwen_vl_utils` composites alpha. The shared `load_image_item` takes the conversion by name; legacy processors keep `smart_rgb_conversion` and their behavior. The two keys are rejected without `--enable-token-space-processor`, since no legacy loader reads them. Signed-off-by: rockdu <kangrdu@gmail.com>
…ent time and reject them where no strategy reads them `handle_multimodal` resolves `image_decoder` / `image_rgb_conversion` with the token-space processor on, so a wrong value fails at argument resolution instead of when the tokenizer manager builds the processor; both resolvers take JSON null as the default and reject a non-string with the same ValueError. `get_mm_processor_cls` raises when the flag is on but the model is served by a legacy processor, which never reads the keys. Signed-off-by: rockdu <kangrdu@gmail.com>
…sses, loader options on the IO path, Qwen's white default The failure-classification test raises an undecodable-image error, which only `media_load_errors` turns into a ValueError; the loader-options test runs an RGBA PNG through `TokenSpaceProcessor.process` and checks what `process_image` receives for the strategy's `discard_alpha_channel`, the server's `image_rgb_conversion` and the default; the legacy-defaults parity test compares `rgb_conversion` by name and records Qwen's `white` as the one declared divergence, so reverting it fails. Signed-off-by: rockdu <kangrdu@gmail.com>
…nd ruff Signed-off-by: rockdu <kangrdu@gmail.com>
…ck so the CI runner executes it The registered CI runner runs a test file as a script, and `unittest.main()` exits before a class defined below it is created; pytest collected the class, which is why a local pytest run was green. Signed-off-by: rockdu <kangrdu@gmail.com>
…mit keeps only for the legacy mixin signature Signed-off-by: rockdu <kangrdu@gmail.com>
Rockdu
force-pushed
the
rockdu-dev-mm-v2/16-image-load-options
branch
from
November 7, 2026 23:01
159d6de to
ee95628
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Token-space processor only. The legacy processors, the EPD receiver and the embedded Rust MM pipeline keep their current media loading.
load_image_item,load_video_item,load_audio_item, withmedia_load_errorsmapping loader exceptions to client errors). The legacy mixin calls them with the options it had before (rgb_conversion="smart" if cls.smart_rgb_conversion else "convert"), so its behavior does not change.TokenSpaceProcessStrategyloads its own media throughload_image,load_videoandload_audio. The token-space processor routes every submit path through them, so a model's strategy can override how its media are loaded.--mm-process-configkeys (next tovideo_preprocessing_device):image_decoder:pil,nvjpegornvjpeg_fancy, the JPEG decoder (gpu_image_decode).image_rgb_conversion:convert,whiteorsmart, how an image with alpha becomes RGB.whitecomposites onto a white background exactly asqwen_vl_utilsdoes (only for modeRGBA);smartis the GLM heuristic the legacy processors callsmart_rgb_conversion.QwenTokenSpaceProcessStrategysetsrgb_conversion = "white": the Qwen-VL family was trained onqwen_vl_utilspixels, which composite alpha onto white; the legacy Qwen processor keepsconvert("RGB").serving_hook), rejected without--enable-token-space-processor, and rejected when the model is served by a legacy processor, so a key no code reads cannot be set silently.Why
A training framework that replays the engine's preprocessing on the trainer side needs the same pixels on both sides. The engine decoded JPEGs with nvJPEG and dropped alpha with
convert("RGB"), whileqwen_vl_utilsdecodes with PIL and composites RGBA onto white; both were class attributes with no server-level override. Putting the loaders on the strategy makes the per-model choice explicit and overridable per deployment, without touching the legacy processors' code paths.The keys are top-level because the
image/video/audiosub-dicts are forwarded verbatim as the HuggingFace processor'simages_kwargs/videos_kwargs/audio_kwargs; a loader option insideimagewould reach the HF image processor and fail its kwargs validation (transformers 5.x raises on unknown keys).Behavior changes
smart_rgb_conversionkeeps its name and meaning; the extracted loaders receive the same arguments the mixin passed before.convert("RGB"). RGB images (every JPEG) are unaffected; the geo3k recipes are not affected.mm_process_config["image_decoder"]/["image_rgb_conversion"]are new keys; setting them without the token-space processor, or for a model a legacy processor serves, is an error._submit_mm_data_loading_tasks_simple(audio_sample_rate,discard_alpha_channel) only mirror the legacy mixin's signature; the strategy holds both. Marked with a TODO to drop with that signature.Validation
CPU unit tests (no model, no server), both registered in
base-a-test-cpu:test_image_load_options.py(new): the resolvers accept the three decoders and three conversions and reject anything else,whitecomposites only an alpha channel (RGBA on white;LAand palette PNGs with atRNSentry go throughconvert("RGB"), as inqwen_vl_utils), the loader options reach the IO path, and a corrupt image surfaces as a client error (UnidentifiedImageError).test_mm_token_expansion.py:TestStrategyLoaderspins the strategy loader contract (per-strategy defaults, Qwenwhite, config override, real RGBA file through the token-space path), defined before the__main__block so the CI runner executes it.End to end (radixark/miles, colocated RL): an Inkling-Small LoRA audio run, 50 rollouts, with the sglang-miles twin of this branch (#48953, same change on sglang-miles f8d0201) plus the Inkling token-space strategy as the engine; train/rollout logprob abs diff 0.013–0.015 throughout, matching the reference run on the previous engine, held-out AVQA 55.4 → 71.7 (reference 55.1 → 72.8), MMAU 78.3 → 77.3 (reference 78.9 → 76.9).
Landing
Stacked on #15 of the token-space series (
rockdu-dev-mm-v2/15-item-feature-cache-write-once). The same change forsglang-milesis #48953; the eight commits are identical apart from the base. The Inkling token-space strategy builds on this change and follows in its own PR.CI States
Latest PR Test (Base): ❌ Run #42423323347
Latest PR Test (Extra): ❌ Run #42423323263
Latest PR Test (AMD ROCm 10): ❌ Run #42423323399
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: