model(vlm): pixtral

[KivenChen]

Motivation

This PR is a proposal to add support for Pixtral (#2351), and removes blockers to accelerate future Mistral multimodal integrations, such as Mistral 3.1 Small. Feedback and suggestions for improvements are welcome and appreciated.

Modifications

• PixtralVisionModel – a leading vision model from Mistral AI, also the multimodal backbone of Mistral 3.1. Accelerated by VisionAttention and MultimodalInputs.
• LlavaForConditionalGeneration – a flexible text‑vision backbone architecture compatible with pixtral‑12b and other multimodal models. Automatically determine text and vision architecture from provided config.
• PixtralProcessor as well as LlavaMultimodalProcessor, which auto-loads processor for any LlavaForConditionalGeneration with vision config.
• Remove unrelated code from own use case.
• Minor refactor to mm_utils to support multiple, arbitary images in a single request.
• test_generation_models updated to bring in latest models.
• Examples, documentation, and benchmarks
• --disable-multimodal added to server args. Multimodal is enabled by default.
• now chunked prefill is enabled for VLMs by default. Tested multi-image input on qwen2.5VL, llama4 and pixtral, and so far no issues identified.

Functionality Tests

• text generation
• "image" modalities
• "multi-images" modalities (NOTE: functional and normal generation outputs, but issues in prefill chunk layer. Models may omit previously cached mm item, giving answers that miscount images or mistake input images for previous added ones).

Preliminary Benchmark

peak decode throughput: 140-150 tps for all modalities
cuda device: A100x1
vram: 40gb
tested modalities: text / image / multi-image
server args: "REQUEST_TIMEOUT=15 python3 -m sglang.launch_server --model-path mistral-community/pixtral-12b --mem-fraction-static 0.7 --dtype bfloat16"

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

[KivenChen]

multi-image modality has support in the model implementation, but is yet to be integrated to multi-modal data pipelines.(#4754). My private use case branch has a minor refactor about it, but it doesn’t touch the scheduler layer; therefore potential problems are unknown.

Shall I remove functionality tests for multi-image support for now?

[zhaochenyang20]

I will ask yi, mick and yuhao to review this.

[yhyang201]

1. I tried running the modified http_pixtral_generation_test.py, but encountered the following error, which might need to be addressed:

[2025-04-07 08:16:52 TP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 2013, in run_scheduler_process
    scheduler.event_loop_normal()
  File "/root/.python/sglang/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 598, in event_loop_normal
    result = self.run_batch(batch)
  File "/sgl-workspace/sglang/python/sglang/srt/managers/scheduler.py", line 1385, in run_batch
    logits_output, next_token_ids = self.tp_worker.forward_batch_generation(
  File "/sgl-workspace/sglang/python/sglang/srt/managers/tp_worker.py", line 175, in forward_batch_generation
    logits_output = self.model_runner.forward(forward_batch)
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 1000, in forward
    return self.forward_extend(
  File "/sgl-workspace/sglang/python/sglang/srt/model_executor/model_runner.py", line 961, in forward_extend
    return self.model.forward(
  File "/sgl-workspace/sglang/python/sglang/srt/models/llava.py", line 749, in forward
    hidden_states = general_mm_embed_routine(
  File "/sgl-workspace/sglang/python/sglang/srt/managers/mm_utils.py", line 352, in general_mm_embed_routine
    inputs_embeds = embed_mm_inputs(
  File "/sgl-workspace/sglang/python/sglang/srt/managers/mm_utils.py", line 261, in embed_mm_inputs
    embedding, mask = get_embedding_and_mask(
  File "/sgl-workspace/sglang/python/sglang/srt/managers/mm_utils.py", line 136, in get_embedding_and_mask
    embeddings = data_embedding_func(embedding_items)
  File "/sgl-workspace/sglang/python/sglang/srt/models/llava.py", line 736, in get_image_feature
    return self.encode_images(pixel_values)
  File "/sgl-workspace/sglang/python/sglang/srt/models/llava.py", line 133, in encode_images
    image_outputs = self.vision_tower(pixel_values, output_hidden_states=True)
  File "/root/.python/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/.python/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/sgl-workspace/sglang/python/sglang/srt/models/pixtral.py", line 497, in forward
    patch_embeds_list = [
  File "/sgl-workspace/sglang/python/sglang/srt/models/pixtral.py", line 498, in <listcomp>
    self.patch_conv(img.unsqueeze(0).to(self.dtype).to(self.device))
  File "/root/.python/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/.python/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/.python/sglang/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 554, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/root/.python/sglang/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 549, in _conv_forward
    return F.conv2d(
RuntimeError: Expected 3D (unbatched) or 4D (batched) input to conv2d, but got input of size: [1, 2, 3, 400, 400]

2. Also, I noticed that there are some debugging comments and print statements in python/sglang/srt/models/llava.py. Should these be removed?

[KivenChen]

Hi @yhyang201 , I just pushed the final commits that resolves the two problems. The PR statement has also been updated to reflect additional modifications, especially multimodality support.

[yhyang201]

1. When loading the model, the following error occurred:

  File "/root/sglang/kgl/python/sglang/srt/model_loader/loader.py", line 146, in _initialize_model
    return model_class(
  File "/root/sglang/kgl/python/sglang/srt/models/llava.py", line 718, in __init__
    vision_model_cls = self._get_sgl_model_cls(config.vision_config, AutoModel)
  File "/root/sglang/kgl/python/sglang/srt/models/llava.py", line 644, in _get_sgl_model_cls
    raise ValueError(
ValueError: AutoModel found a corresponding model `PixtralVisionModel` for config class `PixtralVisionConfig`, but failed to load it from SGLang ModelRegistry.

Could you help fix this issue?

2. Could you also run the MMMU benchmark test?
   Link: https://github.com/sgl-project/sglang/tree/main/benchmark/mmmu

[KivenChen]

1. When loading the model, the following error occurred:

  File "/root/sglang/kgl/python/sglang/srt/model_loader/loader.py", line 146, in _initialize_model

    return model_class(

  File "/root/sglang/kgl/python/sglang/srt/models/llava.py", line 718, in __init__

    vision_model_cls = self._get_sgl_model_cls(config.vision_config, AutoModel)

  File "/root/sglang/kgl/python/sglang/srt/models/llava.py", line 644, in _get_sgl_model_cls

    raise ValueError(

ValueError: AutoModel found a corresponding model `PixtralVisionModel` for config class `PixtralVisionConfig`, but failed to load it from SGLang ModelRegistry.

Could you help fix this issue?

2. Could you also run the MMMU benchmark test?

Link: https://github.com/sgl-project/sglang/tree/main/benchmark/mmmu

Will share benchmark after completion.

As for automodel issue, it should now be fixed. I believe it was caused by incorrect or circular imports. This PR involves dynamic model loading, so I'll keep testing.

[KivenChen]

Generation Model Test & MMMU results . @zhaochenyang20 @yhyang201

MMMU

pixtral-12b scored around 44 with direct prompt; but Mistral-AI claimed that its MMMU score goes up to 50.9 with CoT prompt.

Note: I extended the original mmmu/bench_sglang to optionally call /generate for benchmark, b/c pixtral is a base model. Moreover, a known issue (#3304) was encountered if mistral chat templates were applied. This change is pushed to this branch.

Benchmark Details

python3 -m sglang.launch_server --model-path /workspace/.cache/hub/mistral-community/pixtral-12b --mem-fraction-static 0.75 --tp-size 2 --dtype bfloat16 --log-level debug --log-requests --log-requests-level 2 --cuda-graph-max-bs 32 

# cuda device: A100 x 2. `tp=2` to support large images.
skipping 15 samples with large images, 1.67% of dataset
samples have been prepared
Processing samples: 100%|█████████████████████████████████████████████| 885/885 [04:11<00:00,  3.52it/s]
Benchmark time: 251.45550560951233
answers saved to: ./val_sglang.json
Evaluating...
{'Accounting': {'acc': 0.3, 'num': 30},
 'Agriculture': {'acc': 0.312, 'num': 16},
 'Architecture_and_Engineering': {'acc': 0.267, 'num': 30},
 'Art': {'acc': 0.533, 'num': 30},
 'Art_Theory': {'acc': 0.633, 'num': 30},
 'Basic_Medical_Science': {'acc': 0.433, 'num': 30},
 'Biology': {'acc': 0.467, 'num': 30},
 'Chemistry': {'acc': 0.3, 'num': 30},
 'Clinical_Medicine': {'acc': 0.433, 'num': 30},
 'Computer_Science': {'acc': 0.4, 'num': 30},
 'Design': {'acc': 0.733, 'num': 30},
 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.367, 'num': 30},
 'Economics': {'acc': 0.3, 'num': 30},
 'Electronics': {'acc': 0.333, 'num': 30},
 'Energy_and_Power': {'acc': 0.467, 'num': 30},
 'Finance': {'acc': 0.333, 'num': 30},
 'Geography': {'acc': 0.433, 'num': 30},
 'History': {'acc': 0.667, 'num': 30},
 'Literature': {'acc': 0.759, 'num': 29},
 'Manage': {'acc': 0.433, 'num': 30},
 'Marketing': {'acc': 0.433, 'num': 30},
 'Materials': {'acc': 0.333, 'num': 30},
 'Math': {'acc': 0.467, 'num': 30},
 'Mechanical_Engineering': {'acc': 0.3, 'num': 30},
 'Music': {'acc': 0.4, 'num': 30},
 'Overall': {'acc': 0.438, 'num': 885},
 'Overall-Art and Design': {'acc': 0.575, 'num': 120},
 'Overall-Business': {'acc': 0.36, 'num': 150},
 'Overall-Health and Medicine': {'acc': 0.427, 'num': 150},
 'Overall-Humanities and Social Science': {'acc': 0.605, 'num': 119},
 'Overall-Science': {'acc': 0.407, 'num': 150},
 'Overall-Tech and Engineering': {'acc': 0.347, 'num': 196},
 'Pharmacy': {'acc': 0.4, 'num': 30},
 'Physics': {'acc': 0.367, 'num': 30},
 'Psychology': {'acc': 0.367, 'num': 30},
 'Public_Health': {'acc': 0.5, 'num': 30},
 'Sociology': {'acc': 0.633, 'num': 30}}
eval out saved to ./val_sglang.json
Overall accuracy: 0.438

Generation Model Test

It turns out more flaky than expected. Some default parameter sets didn't run on a A100 x 2 setup. This PR didn't touch the ops layer; not sure what I can help.

Test Detail

MODELS=("Qwen/Qwen2-1.5B" "Qwen/Qwen2.5-14B-Instruct" "HuggingFaceTB/SmolLM-135M-Instruct" "allenai/OLMo-1B-0724-hf" "THUDM/glm-4-9b-chat" "openai-community/gpt2" "microsoft/Phi-3-small-8k-instruct" "allenai/OLMo-2-1124-7B-Instruct" "ibm-granite/granite-3.0-2b-instruct" "mistral-community/pixtral-12b")
TOTAL=0
PASSED=0
FAILED=0

========================================================
Testing model: Qwen/Qwen2-1.5B
========================================================
Sliding Window Attention is enabled but not implemented for `sdpa`; unexpected results may be encountered.
srt_runner: model_path: /workspace/.cache/hub/Qwen/Qwen2-1.5B
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:02<00:00,  2.13s/it]

hf_outputs.output_strs=[' red. Banana is Yellow. Apple is red. Banana is Yellow. Apple is red. Banana is Yellow. Apple is red. Banana is Yellow. Apple is', ' ______.\nA. London\nB. Edinburgh\nC. Manchester\nD. Glasgow\n答案:\nA\n\n下列关于我国古代文学常识的表述，不', ' it. I have a lot of things to do today. I have to go to the bank to get my money. I have to go to the post office', ' the development of intelligent machines that can perform tasks that typically require human intelligence, such as visual perception, speech recognition, decision-making, and language translation. AI is', 'The report provides a comparative analysis of five proposed congressional advisory commissions that would investigate various aspects of the COVID-19 pandemic. The five proposed commissions are found in']
srt_outputs.output_strs=[' red. Banana is Yellow. Apple is red. Banana is Yellow. Apple is red. Banana is Yellow. Apple is red. Banana is Yellow. Apple is', ' ______.\nA. London\nB. Edinburgh\nC. Manchester\nD. Glasgow\n答案:\nA\n\n下列关于我国古代文学常识的表述，不', ' it. I have a lot of things to do today. I have to go to the bank to get my money. I have to go to the post office', ' the development of intelligent machines that can perform tasks that typically require human intelligence, such as visual perception, speech recognition, decision-making, and language translation. AI is', 'The report provides a comparative analysis of five proposed congressional advisory commissions that would investigate various aspects of the COVID-19 pandemic. The five proposed commissions are found in']
rouge_l_scores=[1.0, 1.0, 1.0, 1.0, 1.0]
prefill logprobs max_diff tensor(0.0469)
decode logprobs max_diff tensor(0.0469)
prefill logprobs max_diff tensor(0.0118)
decode logprobs max_diff tensor(0.0312)
prefill logprobs max_diff tensor(0.0130)
decode logprobs max_diff tensor(0.0190)
prefill logprobs max_diff tensor(0.0187)
decode logprobs max_diff tensor(0.0458)
prefill logprobs max_diff tensor(0.0814)
decode logprobs max_diff tensor(0.0312)
.
----------------------------------------------------------------------
Ran 1 test in 36.430s

OK
✅ Test PASSED for Qwen/Qwen2-1.5B
PASSED=1
TOTAL=1

========================================================
Testing model: Qwen/Qwen2.5-14B-Instruct
========================================================
Sliding Window Attention is enabled but not implemented for `sdpa`; unexpected results may be encountered.
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████| 8/8 [00:24<00:00,  3.08s/it]
srt_runner: model_path: /workspace/.cache/hub/Qwen/Qwen2.5-14B-Instruct
Loading safetensors checkpoint shards: 100% Completed | 8/8 [00:14<00:00,  1.79s/it]

hf_outputs.output_strs=[' London, which is also the largest city in the country. The city has a population of around 9 million people and is located in the southeast of England.', ' to go out for a walk. I have 2 hours of free time. If it takes me 15 minutes to get dressed and ready, and I', ' building smart machines capable of performing tasks that typically require human intelligence. AI systems can be trained to recognize patterns, make decisions, and learn from experience, making them']
srt_outputs.output_strs=[' London, which is also the largest city in the country. The city is located in the southeast of England, on the River Thames. London is a global city', " to go out and play. I have a lot of toys to play with, but I can't decide which one to choose. I have a ball, a", ' building smart machines capable of performing tasks that typically require human intelligence. AI is a broad field that encompasses many subfields, including machine learning, natural language processing,']
rouge_l_scores=[0.6910299003322259, 0.5248868778280543, 0.7277227722772277]
E
======================================================================
ERROR: test_others (test_generation_models.TestGenerationModels.test_others)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/workspace/kiv/sglang/python/sglang/srt/utils.py", line 1774, in retry
    return fn()
......
Exception: retry() exceed maximum number of retries.

----------------------------------------------------------------------
Ran 1 test in 77.292s

FAILED (errors=1)
❌ Test FAILED for Qwen/Qwen2.5-14B-Instruct
FAILED=1
TOTAL=2

========================================================
Testing model: HuggingFaceTB/SmolLM-135M-Instruct
========================================================
hf_outputs.output_strs=[' London, which is the largest city in the UK and the financial capital of the country.\n\n**Location:** London is situated in the southeastern part of the', ' to play outside. I\'m so excited to be a part of this adventure!\n\n**Theme:** "Wildlife Encounters"\n\n**Target', ' developing intelligent machines that can learn, reason, and interact with humans. It involves the design, development, testing, and evaluation of intelligent systems that can perform tasks']
srt_outputs.output_strs=[' London, which is the largest city in the UK and the financial capital of the country.\n\n**Location:** London is situated in the southeastern part of the', ' to play outside. I\'m so excited to be a part of this adventure!\n\n**Theme:** "Wildlife Encounters"\n\n**Target', ' developing intelligent machines that can learn, reason, and interact with humans. It involves the design, development, testing, and evaluation of intelligent systems that can perform tasks']
rouge_l_scores=[1.0, 1.0, 1.0]
prefill logprobs max_diff tensor(0.0224)
decode logprobs max_diff tensor(0.0620)
E
======================================================================
ERROR: test_others (test_generation_models.TestGenerationModels.test_others)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/workspace/kiv/sglang/python/sglang/srt/utils.py", line 1774, in retry
    return fn()
           ^^^^
  File "/workspace/kiv/sglang/python/sglang/test/test_utils.py", line 1020, in <lambda>
    lambda: super(CustomTestCase, self)._callTestMethod(method),
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: decode logprobs are not all close with model_path=HuggingFaceTB/SmolLM-135M-Instruct prompts=['The capital of the United Kingdom is', 'Today is a sunny day and I like', 'AI is a field of computer science focused on'] decode_tolerance=0.05.hf_logprobs=tensor([[-3.0960e-01, -2.9737e+00, -4.0674e+00, -4.0987e+00, -4.3487e+00],
        [-4.3457e-01, -1.1377e+00, -4.5908e+00, -4.8564e+00, -5.9814e+00],
        ......
        [-3.0525e-01, -1.4615e+00, -3.8052e+00, -5.9302e+00, -6.1959e+00]]), srt_logprobs=tensor([[-3.0884e-01, -2.9729e+00, -4.0667e+00, -4.0979e+00, -4.3557e+00],
        [-4.3459e-01, -1.1377e+00, -4.5908e+00, -4.8565e+00, -5.9971e+00],
        ......
        [-1.2647e-04, -1.0297e+01, -1.0875e+01, -1.1250e+01, -1.1320e+01],
        [-3.0524e-01, -1.4615e+00, -3.8052e+00, -5.9302e+00, -6.1959e+00]])

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workspace/kiv/sglang/python/sglang/test/test_utils.py", line 1019, in _callTestMethod
    retry(
  File "/workspace/kiv/sglang/python/sglang/srt/utils.py", line 1777, in retry
    raise Exception(f"retry() exceed maximum number of retries.")
Exception: retry() exceed maximum number of retries.

----------------------------------------------------------------------
Ran 1 test in 26.392s

FAILED (errors=1)
❌ Test FAILED for HuggingFaceTB/SmolLM-135M-Instruct
FAILED=2
TOTAL=3

========================================================
Testing model: allenai/OLMo-1B-0724-hf
========================================================
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████| 2/2 [00:02<00:00,  1.34s/it]
srt_runner: model_path: /workspace/.cache/hub/allenai/OLMo-1B-0724-hf

hf_outputs.output_strs=[' red. The sky is blue. Grass is green. The sun is yellow. The moon is white. Water is clear. Snow is white.', ' London, which is the capital city of England and the United Kingdom. London is a global city and one of the most important', ' playing with my friends. We can play tag, hide and seek, or maybe even build a fort. I\'m so excited!', ' building machines that can perform tasks that would normally require human intelligence. This includes things like learning, reasoning, problem-solving, perception, and language understanding.', ' provides a comprehensive analysis of the current state of the market, including trends, challenges, and opportunities. The report examines the key drivers of market growth and offers insights into the competitive landscape.']
srt_outputs.output_strs=[' red.\n\nThe sky is blue. Grass is green. The sun is yellow. The moon is white. Water is clear. Snow is white.', ' London, which is the capital of England and the United Kingdom. London is a major city with a rich history and diverse', " playing. I love to play games with my friends. My favorite games are hide and seek, tag, and Simon says. I'm", ' building machines that can think, learn, and make decisions like humans. This field encompasses various disciplines, including computer science, mathematics, linguistics, psychology,', ' provides a comprehensive analysis of the current state of the market for XYZ products. The report includes detailed information on market size,']

srt_logprobs = tensor([[-2.2465e-01, -3.5528e+00, -4.0450e+00],
        [-8.3630e-01, -1.3050e+00, -1.8050e+00],
        [-1.5826e+00, -1.8482e+00, -2.1529e+00],
......
        [-1.9357e-01, -2.2873e+00, -3.7717e+00],
        [-2.4740e-01, -2.0286e+00, -3.4193e+00]])

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workspace/kiv/sglang/python/sglang/test/test_utils.py", line 1019, in _callTestMethod
    retry(
......
Exception: retry() exceed maximum number of retries.

----------------------------------------------------------------------
Ran 1 test in 28.605s

FAILED (errors=1)
❌ Test FAILED for allenai/OLMo-1B-0724-hf
FAILED=3
TOTAL=4

========================================================
Testing model: THUDM/glm-4-9b-chat
========================================================
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████| 4/4 [00:18<00:00,  4.58s/it]
srt_runner: model_path: /workspace/.cache/hub/THUDM/glm-4-9b-chat

[2025-04-08 10:10:17 TP1] Scheduler hit an exception: Traceback (most recent call last):
  File "/workspace/kiv/sglang/python/sglang/srt/managers/scheduler.py", line 1999, in run_scheduler_process
    scheduler = Scheduler(server_args, port_args, gpu_id, tp_rank, dp_rank)
......
ValueError: Model architectures ['GlmForCausalLM'] are not supported for now. Supported architectures: dict_keys(['BaichuanForCausalLM', 'ChatGLMModel', 'CLIPModel', 'CohereForCausalLM', 'Cohere2ForCausalLM', 'DbrxForCausalLM', 'DeepseekForCausalLM', 'MultiModalityCausalLM', 'DeepseekV3ForCausalLMNextN', 'DeepseekV2ForCausalLM', 'DeepseekV3ForCausalLM', 'DeepseekVL2ForCausalLM', 'ExaoneForCausalLM', 'GemmaForCausalLM', 'Gemma2ForCausalLM', 'Gemma2ForSequenceClassification', 'Gemma3ForCausalLM', 'Gemma3ForConditionalGeneration', 'GPT2LMHeadModel', 'GPTBigCodeForCausalLM', 'GraniteForCausalLM', 'Grok1ForCausalLM', 'Grok1ModelForCausalLM', 'InternLM2ForCausalLM', 'InternLM2ForRewardModel', 'LlamaForCausalLM', 'Phi3ForCausalLM', 'InternLM3ForCausalLM', 'Llama4ForCausalLM', 'LlamaForClassification', 'LlamaForCausalLMEagle', 'LlamaForCausalLMEagle3', 'LlamaEmbeddingModel', 'MistralModel', 'LlamaForSequenceClassification', 'LlamaForSequenceClassificationWithNormal_Weights', 'LlavaLlamaForCausalLM', 'LlavaQwenForCausalLM', 'LlavaMistralForCausalLM', 'LlavaForConditionalGeneration', 'LlavaVidForCausalLM', 'MiniCPMForCausalLM', 'MiniCPM3ForCausalLM', 'MiniCPMO', 'MiniCPMV', 'MistralForCausalLM', 'MixtralForCausalLM', 'QuantMixtralForCausalLM', 'MllamaForConditionalGeneration', 'Llama4ForConditionalGeneration', 'OlmoForCausalLM', 'Olmo2ForCausalLM', 'OlmoeForCausalLM', 'Phi3SmallForCausalLM', 'PixtralVisionModel', 'QWenLMHeadModel', 'Qwen2ForCausalLM', 'Qwen2_5_VLForConditionalGeneration', 'Qwen2ForSequenceClassification', 'Qwen2ForCausalLMEagle', 'Qwen2MoeForCausalLM', 'Qwen2ForRewardModel', 'Qwen2VLForConditionalGeneration', 'StableLmForCausalLM', 'TorchNativeLlamaForCausalLM', 'TorchNativePhi3ForCausalLM', 'XverseForCausalLM', 'XverseMoeForCausalLM', 'YiVLForCausalLM'])

[2025-04-08 10:10:17] Received sigquit from a child process. It usually means the child failed.
❌ Test FAILED for THUDM/glm-4-9b-chat
FAILED=4
TOTAL=5

========================================================
Testing model: openai-community/gpt2
========================================================
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [348,0,0], thread: [125,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [348,0,0], thread: [126,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
../aten/src/ATen/native/cuda/Indexing.cu:1308: indexSelectLargeIndex: block: [348,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Process Process-1:
Traceback (most recent call last):
  File "/root/miniconda3/envs/kgl/lib/python3.11/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 829, in forward
    attention_mask = _prepare_4d_causal_attention_mask_for_sdpa(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/kgl/lib/python3.11/site-packages/transformers/modeling_attn_mask_utils.py", line 378, in _prepare_4d_causal_attention_mask_for_sdpa
    ignore_causal_mask = AttentionMaskConverter._ignore_causal_mask_sdpa(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/kgl/lib/python3.11/site-packages/transformers/modeling_attn_mask_utils.py", line 288, in _ignore_causal_mask_sdpa
    elif not is_tracing and torch.all(attention_mask == 1):
  File "/root/miniconda3/envs/kgl/lib/python3.11/site-packages/torch/utils/_device.py", line 106, in __torch_function__
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

❌ Test FAILED for microsoft/Phi-3-small-8k-instruct
FAILED=6
TOTAL=7

========================================================
Testing model: allenai/OLMo-2-1124-7B-Instruct
========================================================
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████| 3/3 [00:16<00:00,  5.41s/it]
srt_runner: model_path: /workspace/.cache/hub/allenai/OLMo-2-1124-7B-Instruct
Loading safetensors checkpoint shards: 100% Completed | 3/3 [00:12<00:00,  4.25s/it]

hf_outputs.output_strs=[' London, England. London is the largest city in the UK and one of the most populous cities in Europe. It is also a major global city and financial center', ' to go out for a walk in the park. I usually walk for about 30 minutes, covering a distance of 5 kilometers. Today, I decide to', ' creating intelligent machines that can perform tasks that typically require human intelligence. This includes tasks suchassistant as visual perception, speech recognition, decision-making, and language translation.\n\n']
srt_outputs.output_strs=[' London, England. London is the largest city in the UK and one of the most populous cities in Europe. It is also a major global city and financial center', ' to go out for a walk in the park. I usually walk for about 30 minutes, covering a distance of 5 kilometers. Today, I decide to', ' creating intelligent machines that can perform tasks that typically require human intelligence. This includes tasks suchassistant as visual perception, speech recognition, decision-making, and language translation.\n\n']
rouge_l_scores=[1.0, 1.0, 1.0]
prefill logprobs max_diff tensor(0.0140)
decode logprobs max_diff tensor(0.0290)
prefill logprobs max_diff tensor(0.0153)
decode logprobs max_diff tensor(0.0146)
prefill logprobs max_diff tensor(0.0151)
decode logprobs max_diff tensor(0.0253)
.
----------------------------------------------------------------------
Ran 1 test in 59.573s

OK
✅ Test PASSED for allenai/OLMo-2-1124-7B-Instruct
PASSED=2
TOTAL=8

========================================================
Testing model: ibm-granite/granite-3.0-2b-instruct
========================================================
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████| 2/2 [00:05<00:00,  2.86s/it]
srt_runner: model_path: /workspace/.cache/hub/ibm-granite/granite-3.0-2b-instruct
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:04<00:00,  2.22s/it]

hf_outputs.output_strs=[' London.\n\nThe capital of the United Kingdom is London.\n\nThe capital of the United Kingdom is London.\n\nThe capital of the', ' to go for a walk in the park. I can see the green grass, the colorful flowers, and the beautiful trees. I can hear', ' creating intelligent systems that can understand, learn, and make decisions based on data. Here are some key aspects of AI:\n\n1. **Machine Learning']
srt_outputs.output_strs=[' London.\n\nThe capital of the United Kingdom is London.\n\nThe capital of the United Kingdom is London.\n\nThe capital of the', ' to go for a walk in the park. I can see the green grass, the colorful flowers, and the beautiful trees. I can hear', ' creating intelligent systems that can understand, learn, and make decisions based on data. Here are some key aspects of AI:\n\n1. **Machine Learning']
rouge_l_scores=[1.0, 1.0, 1.0]
prefill logprobs max_diff tensor(0.0207)
decode logprobs max_diff tensor(0.0312)
prefill logprobs max_diff tensor(0.0156)
decode logprobs max_diff tensor(0.0312)
prefill logprobs max_diff tensor(0.0209)
decode logprobs max_diff tensor(0.0303)
.
----------------------------------------------------------------------
Ran 1 test in 38.153s

OK
✅ Test PASSED for ibm-granite/granite-3.0-2b-instruct
PASSED=3
TOTAL=9

The last one is tested separately - mistral-community/pixtral-12b

Loading checkpoint shards: 100%|██████████████████████████████████████████████████████| 6/6 [00:16<00:00,  2.74s/it]
srt_runner: model_path: /workspace/.cache/hub/mistral-community/pixtral-12b
Loading safetensors checkpoint shards: 100% Completed | 6/6 [00:12<00:00,  2.02s/it]

hf_outputs.output_strs=[' London. It is the largest city in the UK and is located in the southeast of England. London is a global city and has been a major center of finance', ' to go out and enjoy the sun. I like to go to the park and play with my friends. I like to go to the playground and play on the', ' creating intelligent machines that can perform tasks that typically require human intelligence. These tasks include speech recognition, decision-making, visual perception, and language translation, among others.\n\n']
srt_outputs.output_strs=[' London. It is the largest city in the UK and is located in the southeast of England. London is a global city and has been a major center of finance', ' to go out and enjoy the sun. I like to go to the park and play with my friends. I like to go to the playground and play on the', ' creating intelligent machines that can perform tasks that typically require human intelligence. These tasks include speech recognition, decision-making, visual perception, and language translation, among others.\n\n']
rouge_l_scores=[1.0, 1.0, 1.0]
prefill logprobs max_diff tensor(0.0199)
decode logprobs max_diff tensor(0.0291)
prefill logprobs max_diff tensor(0.0127)
decode logprobs max_diff tensor(0.0174)
prefill logprobs max_diff tensor(0.0155)
decode logprobs max_diff tensor(0.0312)
.
----------------------------------------------------------------------
Ran 1 test in 68.452s

OK

[zhaochenyang20]

cc @yhyang201 could you take a look plz?

[KivenChen]

multimodal part just aligned

[JustinTong0323]

MMMU bench result:

{'Accounting': {'acc': 0.4, 'num': 30},
 'Agriculture': {'acc': 0.562, 'num': 16},
 'Architecture_and_Engineering': {'acc': 0.333, 'num': 30},
 'Art': {'acc': 0.667, 'num': 30},
 'Art_Theory': {'acc': 0.7, 'num': 30},
 'Basic_Medical_Science': {'acc': 0.467, 'num': 30},
 'Biology': {'acc': 0.5, 'num': 30},
 'Chemistry': {'acc': 0.3, 'num': 30},
 'Clinical_Medicine': {'acc': 0.5, 'num': 30},
 'Computer_Science': {'acc': 0.4, 'num': 30},
 'Design': {'acc': 0.6, 'num': 30},
 'Diagnostics_and_Laboratory_Medicine': {'acc': 0.467, 'num': 30},
 'Economics': {'acc': 0.4, 'num': 30},
 'Electronics': {'acc': 0.267, 'num': 30},
 'Energy_and_Power': {'acc': 0.433, 'num': 30},
 'Finance': {'acc': 0.233, 'num': 30},
 'Geography': {'acc': 0.4, 'num': 30},
 'History': {'acc': 0.8, 'num': 30},
 'Literature': {'acc': 0.862, 'num': 29},
 'Manage': {'acc': 0.4, 'num': 30},
 'Marketing': {'acc': 0.4, 'num': 30},
 'Materials': {'acc': 0.233, 'num': 30},
 'Math': {'acc': 0.4, 'num': 30},
 'Mechanical_Engineering': {'acc': 0.3, 'num': 30},
 'Music': {'acc': 0.433, 'num': 30},
 'Overall': {'acc': 0.463, 'num': 885},
 'Overall-Art and Design': {'acc': 0.6, 'num': 120},
 'Overall-Business': {'acc': 0.367, 'num': 150},
 'Overall-Health and Medicine': {'acc': 0.473, 'num': 150},
 'Overall-Humanities and Social Science': {'acc': 0.706, 'num': 119},
 'Overall-Science': {'acc': 0.4, 'num': 150},
 'Overall-Tech and Engineering': {'acc': 0.347, 'num': 196},
 'Pharmacy': {'acc': 0.467, 'num': 30},
 'Physics': {'acc': 0.4, 'num': 30},
 'Psychology': {'acc': 0.6, 'num': 30},
 'Public_Health': {'acc': 0.467, 'num': 30},
 'Sociology': {'acc': 0.567, 'num': 30}}
eval out saved to ./val_sglang.json
Overall accuracy: 0.463

[zhaochenyang20]

great work！