[Bug] Gemma-3-27B-it outputs unrelated nonsense

[RodriMora]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

When prompting Gemma3-27B-it it outputs and unrelated answer to the prompt

Reproduction

python3 -m venv .venv
source .venv/bin/activate.fish
pip install --upgrade pip
pip install uv
uv pip install "sglang[all]>=0.4.5" --find-links https://flashinfer.ai/whl/cu124/torch2.5/flashinfer-python

CUDA_VISIBLE_DEVICES=0,2,3,4 python -m sglang.launch_server --model-path /mnt/llms/models/gemma-3-27b-it --tp 4 --port 5000 --host 0.0.0.0 --context-length 25000

curl -X POST "http://192.168.10.115:5000/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "/mnt/llms/models/gemma-3-27b-it",
    "messages": [
      {
        "role": "user",
        "content": "write python code to check the price of BTC"
      }
    ],
    "temperature": 0.8,
    "max_tokens": 4096
  }'

Environment

Ubuntu 22.04
Python3.10 using venv
CPU: Epyc 7402
Motherboard: Supermicro H12SSL-i
RAM: 512GB RAM
GPUs:
Device 0 = 3090
Device 1 = 5090
Device 2 = 3090
Device 3 = 3090
Device 4 = 3090

The 5090 is in slot pcie 1 (the top one) but is recognised as the device 1 instead of 0, not sure why. I have CUDA_DEVICE_ORDER=PCI_BUS_ID set.
I'm using CUDA_VISIBLE_DEVICES=0,2,3,4 so the 5090 is not used as I believe is not supported yet

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Fri_Feb_21_20:23:50_PST_2025
Cuda compilation tools, release 12.8, V12.8.93
Build cuda_12.8.r12.8/compiler.35583870_0

python -m pip list
Package                   Version
------------------------- -------------------
aiohappyeyeballs          2.6.1
aiohttp                   3.11.16
aiosignal                 1.3.2
airportsdata              20250224
annotated-types           0.7.0
anthropic                 0.49.0
anyio                     4.9.0
asttokens                 3.0.0
async-timeout             5.0.1
attrs                     25.3.0
certifi                   2025.1.31
cffi                      1.17.1
charset-normalizer        3.4.1
click                     8.1.8
cloudpickle               3.1.1
compressed-tensors        0.9.3
cuda-bindings             12.8.0
cuda-python               12.8.0
datasets                  3.5.0
decorator                 5.2.1
decord                    0.6.0
dill                      0.3.8
diskcache                 5.6.3
distro                    1.9.0
einops                    0.8.1
exceptiongroup            1.2.2
executing                 2.2.0
fastapi                   0.115.12
filelock                  3.18.0
flashinfer-python         0.2.3+cu124torch2.5
frozenlist                1.5.0
fsspec                    2024.12.0
h11                       0.14.0
hf_transfer               0.1.9
httpcore                  1.0.7
httpx                     0.28.1
huggingface-hub           0.30.1
idna                      3.10
importlib_metadata        8.6.1
interegular               0.3.3
ipython                   8.34.0
jedi                      0.19.2
Jinja2                    3.1.6
jiter                     0.9.0
jsonschema                4.23.0
jsonschema-specifications 2024.10.1
lark                      1.2.2
litellm                   1.65.4.post1
llguidance                0.7.13
MarkupSafe                3.0.2
matplotlib-inline         0.1.7
modelscope                1.24.1
mpmath                    1.3.0
multidict                 6.3.2
multiprocess              0.70.16
nanobind                  2.6.1
nest-asyncio              1.6.0
networkx                  3.4.2
ninja                     1.11.1.4
numpy                     2.2.4
nvidia-cublas-cu12        12.4.5.8
nvidia-cuda-cupti-cu12    12.4.127
nvidia-cuda-nvrtc-cu12    12.4.127
nvidia-cuda-runtime-cu12  12.4.127
nvidia-cudnn-cu12         9.1.0.70
nvidia-cufft-cu12         11.2.1.3
nvidia-curand-cu12        10.3.5.147
nvidia-cusolver-cu12      11.6.1.9
nvidia-cusparse-cu12      12.3.1.170
nvidia-ml-py              12.570.86
nvidia-nccl-cu12          2.21.5
nvidia-nvjitlink-cu12     12.4.127
nvidia-nvtx-cu12          12.4.127
openai                    1.70.0
orjson                    3.10.16
outlines                  0.1.11
outlines_core             0.1.26
packaging                 24.2
pandas                    2.2.3
parso                     0.8.4
partial-json-parser       0.2.1.1.post5
pexpect                   4.9.0
pillow                    11.1.0
pip                       22.0.2
prometheus_client         0.21.1
prompt_toolkit            3.0.50
propcache                 0.3.1
psutil                    7.0.0
ptyprocess                0.7.0
pure_eval                 0.2.3
pyarrow                   19.0.1
pycountry                 24.6.1
pycparser                 2.22
pydantic                  2.11.2
pydantic_core             2.33.1
Pygments                  2.19.1
pynvml                    12.0.0
python-dateutil           2.9.0.post0
python-dotenv             1.1.0
python-multipart          0.0.20
pytz                      2025.2
PyYAML                    6.0.2
pyzmq                     26.4.0
referencing               0.36.2
regex                     2024.11.6
requests                  2.32.3
rpds-py                   0.24.0
safetensors               0.5.3
sentencepiece             0.2.0
setproctitle              1.3.5
setuptools                59.6.0
sgl-kernel                0.0.8
sglang                    0.4.5
six                       1.17.0
sniffio                   1.3.1
soundfile                 0.13.1
stack-data                0.6.3
starlette                 0.46.1
sympy                     1.13.1
tiktoken                  0.9.0
tokenizers                0.21.1
torch                     2.5.1
torchao                   0.9.0
tqdm                      4.67.1
traitlets                 5.14.3
transformers              4.51.0
triton                    3.1.0
typing_extensions         4.13.1
typing-inspection         0.4.0
tzdata                    2025.2
urllib3                   2.3.0
uv                        0.6.12
uvicorn                   0.34.0
uvloop                    0.21.0
wcwidth                   0.2.13
xgrammar                  0.1.17
xxhash                    3.5.0
yarl                      1.19.0
zipp                      3.21.0

Output:

curl -X POST "http://192.168.10.115:5000/v1/chat/completions" \
                          -H "Content-Type: application/json" \
                          -d '{
                        "model": "/mnt/llms/models/gemma-3-27b-it",
                        "messages": [
                          {
                            "role": "user",
                            "content": "write python code to check the price of BTC"
                          }
                        ],
                        "temperature": 0.8,
                        "max_tokens": 4096
                      }'
{"id":"5b109e7c2cd2440d875a635ee6e9ab99","object":"chat.completion","created":1744017128,"model":"/mnt/llms/models/gemma-3-27b-it","choices":[{"index":0,"message":{"role":"assistant","content":"Tali Capung (Dragonfly String) is a type of traditional Indonesian handicraft made from woven fibers, creating a delicate and intricate string. It's known for its vibrant colors and decorative designs, and it's often used in a variety of applications. Here's a breakdown of what you need to know about Tali Capung:\n\n**1. Materials and Production:**\n\n*   **Fiber Source:** Traditionally, Tali Capung is made from the fibers of the *Ronca* plant (also known as *Cabuk*), a type of shrub common in Indonesia, particularly in Java. Newer versions may utilize synthetic fibers, but the authentic kind focuses on the natural material.\n*   **Processing:** The Ronca plant stalks are harvested, the fibers are extracted, dyed with natural or synthetic dyes to achieve vibrant colors, and then meticulously woven together.  This weaving is done by hand, making each piece unique. The fibers are very thin and require skill to work with.\n*   **\"Capung\" (Dragonfly) Name:** The name \"Tali Capung\" comes from the resemblance of the finished string to the delicate wings of a dragonfly. The intricate weaving creates a light, airy texture.\n\n**2. Uses & Applications:**\n\n*   **Decoration:** This is the most common use. Tali Capung is used to decorate:\n    *   **Batik:**  It's sewn onto Batik clothing, bags, and other textiles as an embellishment.  It's very popular to add edging or motifs.\n    *   **Home Decor:**  Used for trim on cushions, curtains, tablecloths, wall hangings, and other decorative items.\n    *   **Gift Wrapping:** Tali Capung adds a beautiful, handcrafted touch to gift packaging.\n    *   **Accessories:**  Used in jewelry making (bracelets, necklaces, earrings), keychains, and embellishments for hats and clothing.\n*   **Traditional Ceremonies:**  In some areas, Tali Capung is used in traditional ceremonies and rituals as a decorative element.\n*   **Fashion:**  Designers incorporate Tali Capung into clothing designs, adding a unique Indonesian element to their creations.\n*   **Craft Projects:** A popular material for crafting and DIY projects.\n\n**3. Characteristics:**\n\n*   **Delicate & Lightweight:** Tali Capung is very thin and light.\n*   **Vibrant Colors:**  It comes in a wide range of bright and saturated colors.\n*   **Intricate Designs:** The weaving creates a variety of patterns and textures.\n*   **Handmade:**  Being handmade, variations in color and pattern are common and contribute to its unique charm.\n*   **Texture:**  It has a slightly rough, natural texture.\n*   **Durability:** While delicate, when cared for properly, it can be quite durable.\n\n\n\n**4. Where to Find It:**\n\n*   **Indonesia:**  You will find Tali Capung in traditional markets (pasar tradisional), craft shops, and Batik shops throughout Indonesia, particularly in Java.\n*   **Online Marketplaces:**  Etsy, eBay, and specialized Indonesian handicraft websites sell Tali Capung.\n*   **Indonesian Shops Abroad:**  Some Indonesian stores located in other countries may carry it.\n*   **Craft Fairs:**  Look for vendors specializing in Indonesian crafts at craft fairs.\n\n\n\n**5. Care Instructions:**\n\n*   **Hand Wash:**  Gentle hand washing is recommended.  Avoid harsh detergents or bleach.\n*   **Air Dry:**  Lay flat to air dry. Avoid direct sunlight, which can fade the colors.\n*   **Ironing:**  Iron on a low setting, if necessary, with a pressing cloth.\n*   **Storage:** Store in a cool, dry place to prevent mildew or damage.\n\n\n\nTali Capung is a beautiful example of Indonesian artistry and a versatile material that adds a touch of culture and handcrafted charm to any project or item.  Its delicate beauty and vibrant colors make it a popular choice for those seeking unique and authentic Indonesian crafts.","reasoning_content":null,"tool_calls":null},"logprobs":null,"finish_reason":"stop","matched_stop":106}],"usage":{"prompt_tokens":1,"total_tokens":857,"completion_tokens":856,"prompt_tokens_details":null}}

[sld]

Same for me on 2x4090.

docker-compose.yml:

services:
  llm:
    image: lmsysorg/sglang:v0.4.5-cu125
    ulimits:
      memlock: -1
      stack: 67108864
    ipc: host
    entrypoint: python3 -m sglang.launch_server
    command: >
      --model /llm_models/google_gemma-3-27b-it
      --host 0.0.0.0
      --port 80
      --log-requests
      --random-seed 42
      --grammar-backend xgrammar
      --enable-metrics
      --tp 2
      --enable-p2p-check
      --mem-fraction-static 0.73

Sglang logs:

server_args=ServerArgs(model_path='/llm_models/google_gemma-3-27b-it', tokenizer_path='/llm_models/google_gemma-3-27b-it', tokenizer_mode='auto', skip_tokenizer_init=False, load_format='auto', trust_remote_code=False, dtype='auto', kv_cache_dtype='auto', quantization=None, quantization_param_path=None, context_length=None, device='cuda', served_model_name='/llm_models/google_gemma-3-27b-it', chat_template=None, completion_template=None, is_embedding=False, revision=None, host='0.0.0.0', port=80, mem_fraction_static=0.73, max_running_requests=None, max_total_tokens=None, chunked_prefill_size=8192, max_prefill_tokens=16384, schedule_policy='fcfs', schedule_conservativeness=1.0, cpu_offload_gb=0, page_size=1, tp_size=2, stream_interval=1, stream_output=False, random_seed=42, constrained_json_whitespace_pattern=None, watchdog_timeout=300, dist_timeout=None, download_dir=None, base_gpu_id=0, gpu_id_step=1, log_level='info', log_level_http=None, log_requests=True, log_requests_level=0, show_time_cost=False, enable_metrics=True, decode_log_interval=40, api_key=None, file_storage_path='sglang_storage', enable_cache_report=False, reasoning_parser=None, dp_size=1, load_balance_method='round_robin', ep_size=1, dist_init_addr=None, nnodes=1, node_rank=0, json_model_override_args='{}', lora_paths=None, max_loras_per_batch=8, lora_backend='triton', attention_backend=None, sampling_backend='flashinfer', grammar_backend='xgrammar', speculative_algorithm=None, speculative_draft_model_path=None, speculative_num_steps=None, speculative_eagle_topk=None, speculative_num_draft_tokens=None, speculative_accept_threshold_single=1.0, speculative_accept_threshold_acc=1.0, speculative_token_map=None, enable_double_sparsity=False, ds_channel_config_path=None, ds_heavy_channel_num=32, ds_heavy_token_num=256, ds_heavy_channel_type='qk', ds_sparse_decode_threshold=4096, disable_radix_cache=False, disable_cuda_graph=False, disable_cuda_graph_padding=False, enable_nccl_nvls=False, disable_outlines_disk_cache=False, disable_custom_all_reduce=False, disable_mla=False, disable_overlap_schedule=False, enable_mixed_chunk=False, enable_dp_attention=False, enable_ep_moe=False, enable_deepep_moe=False, deepep_mode=None, enable_torch_compile=False, torch_compile_max_bs=32, cuda_graph_max_bs=160, cuda_graph_bs=None, torchao_config='', enable_nan_detection=False, enable_p2p_check=True, triton_attention_reduce_in_fp32=False, triton_attention_num_kv_splits=8, num_continuous_decode_steps=1, delete_ckpt_after_loading=False, enable_memory_saver=False, allow_auto_truncate=False, enable_custom_logit_processor=False, tool_call_parser=None, enable_hierarchical_cache=False, hicache_ratio=2.0, enable_flashinfer_mla=False, enable_flashmla=False, flashinfer_mla_disable_ragged=False, warmups=None, n_share_experts_fusion=0, disable_shared_experts_fusion=False, debug_tensor_dump_output_folder=None, debug_tensor_dump_input_file=None, debug_tensor_dump_inject=False, disaggregation_mode='null', disaggregation_bootstrap_port=8998)
llm-1  | [2025-04-07 12:13:33] For Gemma 3, we downcast float32 to bfloat16 instead of float16 by default. Please specify `dtype` if you want to use float16.
llm-1  | [2025-04-07 12:13:33] Downcasting torch.float32 to torch.bfloat16.
llm-1  | [2025-04-07 12:13:33] Ignore import error when loading sglang.srt.managers.multimodal_processors.mllama4: No module named 'transformers.models.llama4.image_processing_llama4'
llm-1  | Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
llm-1  | [2025-04-07 12:13:37 TP0] For Gemma 3, we downcast float32 to bfloat16 instead of float16 by default. Please specify `dtype` if you want to use float16.
llm-1  | [2025-04-07 12:13:37 TP0] Downcasting torch.float32 to torch.bfloat16.
llm-1  | Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
llm-1  | [2025-04-07 12:13:37 TP1] For Gemma 3, we downcast float32 to bfloat16 instead of float16 by default. Please specify `dtype` if you want to use float16.
llm-1  | [2025-04-07 12:13:37 TP1] Downcasting torch.float32 to torch.bfloat16.
llm-1  | Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.

llm-1  | [2025-04-07 12:13:40 TP0] Overlap scheduler is disabled for multimodal models.
llm-1  | [2025-04-07 12:13:40 TP0] For Gemma 3, we downcast float32 to bfloat16 instead of float16 by default. Please specify `dtype` if you want to use float16.
llm-1  | [2025-04-07 12:13:40 TP0] Downcasting torch.float32 to torch.bfloat16.
llm-1  | [2025-04-07 12:13:40 TP0] Attention backend not set. Use flashinfer backend by default.
llm-1  | [2025-04-07 12:13:40 TP0] Automatically reduce --mem-fraction-static to 0.657 because this is a multimodal model.
llm-1  | [2025-04-07 12:13:40 TP0] Automatically turn off --chunked-prefill-size for multimodal model.
llm-1  | [2025-04-07 12:13:40 TP0] Init torch distributed begin.
llm-1  | [2025-04-07 12:13:40 TP1] Overlap scheduler is disabled for multimodal models.
llm-1  | [2025-04-07 12:13:40 TP1] For Gemma 3, we downcast float32 to bfloat16 instead of float16 by default. Please specify `dtype` if you want to use float16.
llm-1  | [2025-04-07 12:13:40 TP1] Downcasting torch.float32 to torch.bfloat16.
llm-1  | [2025-04-07 12:13:40 TP1] Attention backend not set. Use flashinfer backend by default.
llm-1  | [2025-04-07 12:13:40 TP1] Automatically reduce --mem-fraction-static to 0.657 because this is a multimodal model.
llm-1  | [2025-04-07 12:13:40 TP1] Automatically turn off --chunked-prefill-size for multimodal model.
llm-1  | [2025-04-07 12:13:40 TP1] Init torch distributed begin.
llm-1  | [2025-04-07 12:13:40 TP1] sglang is using nccl==2.21.5
llm-1  | [2025-04-07 12:13:40 TP0] sglang is using nccl==2.21.5
llm-1  | [2025-04-07 12:13:40 TP0] generating GPU P2P access cache in /root/.cache/sglang/gpu_p2p_access_cache_for_0,1.json
llm-1  | [2025-04-07 12:13:46 TP0] reading GPU P2P access cache from /root/.cache/sglang/gpu_p2p_access_cache_for_0,1.json
llm-1  | [2025-04-07 12:13:46 TP1] reading GPU P2P access cache from /root/.cache/sglang/gpu_p2p_access_cache_for_0,1.json
llm-1  | [2025-04-07 12:13:46 TP0] Custom allreduce is disabled because your platform lacks GPU P2P capability or P2P test failed. To silence this warning, specify disable_custom_all_reduce=True explicitly.
llm-1  | [2025-04-07 12:13:46 TP1] Custom allreduce is disabled because your platform lacks GPU P2P capability or P2P test failed. To silence this warning, specify disable_custom_all_reduce=True explicitly.
llm-1  | [2025-04-07 12:13:46 TP1] Init torch distributed ends. mem usage=0.19 GB
llm-1  | [2025-04-07 12:13:46 TP0] Init torch distributed ends. mem usage=0.19 GB
llm-1  | [2025-04-07 12:13:46 TP1] Load weight begin. avail mem=46.82 GB
llm-1  | [2025-04-07 12:13:46 TP0] Load weight begin. avail mem=46.82 GB
Loading safetensors checkpoint shards:   0% Completed | 0/12 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   8% Completed | 1/12 [00:01<00:11,  1.01s/it]
Loading safetensors checkpoint shards:  17% Completed | 2/12 [00:01<00:09,  1.05it/s]
Loading safetensors checkpoint shards:  25% Completed | 3/12 [00:02<00:08,  1.10it/s]
Loading safetensors checkpoint shards:  33% Completed | 4/12 [00:03<00:06,  1.16it/s]
Loading safetensors checkpoint shards:  42% Completed | 5/12 [00:04<00:05,  1.17it/s]
Loading safetensors checkpoint shards:  50% Completed | 6/12 [00:05<00:05,  1.17it/s]
Loading safetensors checkpoint shards:  58% Completed | 7/12 [00:06<00:04,  1.17it/s]
Loading safetensors checkpoint shards:  67% Completed | 8/12 [00:06<00:03,  1.18it/s]
Loading safetensors checkpoint shards:  75% Completed | 9/12 [00:07<00:02,  1.18it/s]
Loading safetensors checkpoint shards:  83% Completed | 10/12 [00:08<00:01,  1.18it/s]
Loading safetensors checkpoint shards:  92% Completed | 11/12 [00:08<00:00,  1.59it/s]
llm-1  | [2025-04-07 12:13:56 TP1] Load weight end. type=Gemma3ForConditionalGeneration, dtype=torch.bfloat16, avail mem=19.26 GB, mem usage=27.56 GB.
Loading safetensors checkpoint shards: 100% Completed | 12/12 [00:09<00:00,  1.47it/s]
Loading safetensors checkpoint shards: 100% Completed | 12/12 [00:09<00:00,  1.25it/s]
llm-1  | 
llm-1  | [2025-04-07 12:13:56 TP0] Load weight end. type=Gemma3ForConditionalGeneration, dtype=torch.bfloat16, avail mem=19.26 GB, mem usage=27.56 GB.
llm-1  | [2025-04-07 12:13:57 TP0] KV Cache is allocated. #tokens: 13525, K size: 1.60 GB, V size: 1.60 GB
llm-1  | [2025-04-07 12:13:57 TP0] Memory pool end. avail mem=7.86 GB
llm-1  | [2025-04-07 12:13:57 TP1] KV Cache is allocated. #tokens: 13525, K size: 1.60 GB, V size: 1.60 GB
llm-1  | [2025-04-07 12:13:57 TP1] Memory pool end. avail mem=7.86 GB
llm-1  | [2025-04-07 12:13:57 TP0] Capture cuda graph begin. This can take up to several minutes. avail mem=7.29 GB
Capturing batches (avail_mem=5.88 GB):   0%|          | 0/23 [00:00<?, ?it/s][2025-04-07 12:13:57 TP1] Capture cuda graph begin. This can take up to several minutes. avail mem=7.29 GB
Capturing batches (avail_mem=2.29 GB): 100%|██████████| 23/23 [00:11<00:00,  1.99it/s]
llm-1  | [2025-04-07 12:14:08 TP0] Capture cuda graph end. Time elapsed: 11.56 s. avail mem=2.20 GB. mem usage=5.09 GB.
llm-1  | [2025-04-07 12:14:08 TP1] Capture cuda graph end. Time elapsed: 11.52 s. avail mem=2.20 GB. mem usage=5.09 GB.
llm-1  | [2025-04-07 12:14:11 TP0] max_total_num_tokens=13525, chunked_prefill_size=-1, max_prefill_tokens=16384, max_running_requests=2049, context_len=1048576
llm-1  | [2025-04-07 12:14:11 TP1] max_total_num_tokens=13525, chunked_prefill_size=-1, max_prefill_tokens=16384, max_running_requests=2049, context_len=1048576
llm-1  | [2025-04-07 12:14:12] INFO:     Started server process [1]
llm-1  | [2025-04-07 12:14:12] INFO:     Waiting for application startup.
llm-1  | [2025-04-07 12:14:12] INFO:     Application startup complete.
llm-1  | [2025-04-07 12:14:12] INFO:     Uvicorn running on http://0.0.0.0:80 (Press CTRL+C to quit)
llm-1  | [2025-04-07 12:14:13] INFO:     127.0.0.1:43640 - "GET /get_model_info HTTP/1.1" 200 OK

llm-1  | [2025-04-07 12:14:28] Receive: obj=GenerateReqInput(sampling_params={'temperature': 1.0, 'max_new_tokens': 1024, 'min_new_tokens': 0, 'stop': ['\n\n\n\n\n\n\n', '[INST]', '[/INST]'], 'stop_token_ids': None, 'top_p': 1.0, 'top_k': -1, 'min_p': 0.0, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'repetition_penalty': 1.0, 'regex': None, 'ebnf': None, 'n': 1, 'no_stop_trim': False, 'ignore_eos': False, 'skip_special_tokens': True, 'json_schema': '{"type": "object", "properties": {"answer": {"description": "A comprehensive and well-articulated response to the query.", "type": "string"}, "text_ids": {"description": "Array containing the IDs of text passages that informed the response.", "type": "array", "items": {"type": "integer"}, "minItems": 0, "maxItems": 10}, "confidence": {"description": "A numerical estimate of confidence in the response\'s accuracy, as a percentage.", "type": "number", "minimum": 0, "maximum": 100}}, "required": ["answer", "text_ids", "confidence"]}'}, rid='ccbc964c262c49df9be2e26a44dd6da7', return_logprob=False, logprob_start_len=-1, top_logprobs_num=0, token_ids_logprob=None, return_text_in_logprobs=True, stream=False, log_metrics=True, modalities=[], session_params=None, custom_logit_processor=None, return_hidden_states=False)
llm-1  | [2025-04-07 12:14:28 TP0] Prefill batch. #new-seq: 1, #new-token: 1, #cached-token: 0, token usage: 0.00, #running-req: 0, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:10 TP0] Decode batch. #running-req: 1, #token: 34, token usage: 0.00, gen throughput (token/s): 0.67, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:12 TP0] Decode batch. #running-req: 1, #token: 74, token usage: 0.01, gen throughput (token/s): 25.59, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:13 TP0] Decode batch. #running-req: 1, #token: 114, token usage: 0.01, gen throughput (token/s): 25.62, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:15 TP0] Decode batch. #running-req: 1, #token: 154, token usage: 0.01, gen throughput (token/s): 25.62, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:16 TP0] Decode batch. #running-req: 1, #token: 194, token usage: 0.01, gen throughput (token/s): 25.56, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:18 TP0] Decode batch. #running-req: 1, #token: 234, token usage: 0.02, gen throughput (token/s): 25.53, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:20 TP0] Decode batch. #running-req: 1, #token: 274, token usage: 0.02, gen throughput (token/s): 25.52, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:21 TP0] Decode batch. #running-req: 1, #token: 314, token usage: 0.02, gen throughput (token/s): 25.54, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:23 TP0] Decode batch. #running-req: 1, #token: 354, token usage: 0.03, gen throughput (token/s): 25.50, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:24 TP0] Decode batch. #running-req: 1, #token: 394, token usage: 0.03, gen throughput (token/s): 25.51, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:26 TP0] Decode batch. #running-req: 1, #token: 434, token usage: 0.03, gen throughput (token/s): 25.51, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:27 TP0] Decode batch. #running-req: 1, #token: 474, token usage: 0.04, gen throughput (token/s): 25.51, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:29 TP0] Decode batch. #running-req: 1, #token: 514, token usage: 0.04, gen throughput (token/s): 25.63, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:31 TP0] Decode batch. #running-req: 1, #token: 554, token usage: 0.04, gen throughput (token/s): 25.52, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:32 TP0] Decode batch. #running-req: 1, #token: 594, token usage: 0.04, gen throughput (token/s): 25.51, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:34 TP0] Decode batch. #running-req: 1, #token: 634, token usage: 0.05, gen throughput (token/s): 25.54, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:35 TP0] Decode batch. #running-req: 1, #token: 674, token usage: 0.05, gen throughput (token/s): 25.47, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:37 TP0] Decode batch. #running-req: 1, #token: 714, token usage: 0.05, gen throughput (token/s): 25.59, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:38 TP0] Decode batch. #running-req: 1, #token: 754, token usage: 0.06, gen throughput (token/s): 25.52, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:40 TP0] Decode batch. #running-req: 1, #token: 794, token usage: 0.06, gen throughput (token/s): 25.52, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:42 TP0] Decode batch. #running-req: 1, #token: 834, token usage: 0.06, gen throughput (token/s): 25.54, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:43 TP0] Decode batch. #running-req: 1, #token: 874, token usage: 0.06, gen throughput (token/s): 25.50, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:45 TP0] Decode batch. #running-req: 1, #token: 914, token usage: 0.07, gen throughput (token/s): 25.47, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:46 TP0] Decode batch. #running-req: 1, #token: 954, token usage: 0.07, gen throughput (token/s): 25.46, #queue-req: 0, 
llm-1  | [2025-04-07 12:15:48] Finish: obj=GenerateReqInput(sampling_params={'temperature': 1.0, 'max_new_tokens': 1024, 'min_new_tokens': 0, 'stop': ['\n\n\n\n\n\n\n', '[INST]', '[/INST]'], 'stop_token_ids': None, 'top_p': 1.0, 'top_k': -1, 'min_p': 0.0, 'presence_penalty': 0.0, 'frequency_penalty': 0.0, 'repetition_penalty': 1.0, 'regex': None, 'ebnf': None, 'n': 1, 'no_stop_trim': False, 'ignore_eos': False, 'skip_special_tokens': True, 'json_schema': '{"type": "object", "properties": {"answer": {"description": "A comprehensive and well-articulated response to the query.", "type": "string"}, "text_ids": {"description": "Array containing the IDs of text passages that informed the response.", "type": "array", "items": {"type": "integer"}, "minItems": 0, "maxItems": 10}, "confidence": {"description": "A numerical estimate of confidence in the response\'s accuracy, as a percentage.", "type": "number", "minimum": 0, "maximum": 100}}, "required": ["answer", "text_ids", "confidence"]}'}, rid='ccbc964c262c49df9be2e26a44dd6da7', return_logprob=False, logprob_start_len=-1, top_logprobs_num=0, token_ids_logprob=None, return_text_in_logprobs=True, stream=False, log_metrics=True, modalities=[], session_params=None, custom_logit_processor=None, return_hidden_states=False), out={'meta_info': {'id': 'ccbc964c262c49df9be2e26a44dd6da7', 'finish_reason': {'type': 'stop', 'matched': 1}, 'prompt_tokens': 1, 'completion_tokens': 993, 'cached_tokens': 0, 'e2e_latency': 80.12889742851257}}
llm-1  | [2025-04-07 12:15:48] INFO:     10.3.10.83:2272 - "POST /v1/chat/completions HTTP/1.1" 200 OK

On lmsysorg/sglang:v0.4.4.post3-cu125 everything works fine.

[KCFindstr]

Same for me. And even with the v0.4.4.post3 version, the model doesn't seem to recognize vision inputs. This is the chat completions prompt I used:

{
    "model": "gemma-3-27b-it",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/c/cb/The_Blue_Marble_%28remastered%29.jpg/580px-The_Blue_Marble_%28remastered%29.jpg"}},
          {"type": "text", "text": "What is in this image?"}
        ]
      }
    ],
    "max_tokens": 128
  }

[adarshxs]

@RodriMora @KCFindstr @sld can you guys please try launching your server with an extra argument --chat-template gemma-it

[sld]

@RodriMora @KCFindstr @sld can you guys please try launching your server with an extra argument --chat-template gemma-it

With this argument everything works now, thanks!

[RodriMora]

Working good now. Is there any documentation on where to find which models requiere the --chat-template argument?

[adarshxs]

Right now all VLMs require the --chat-template argument. You can find more details here: https://docs.sglang.ai/backend/openai_api_vision.html#Chat-Template

Support for auto setting the chat template is currently in progress, and soon you should be able to skip adding this argument.

[KCFindstr]

Hi @adarshxs, yes it works for the image prompt too! It looks like my original image URL has some problems since it throws this error (although I can open the image from my browser):

  File "/sgl-workspace/sglang/python/sglang/srt/managers/multimodal_processors/gemma3.py", line 43, in process_mm_data_async
    base_output = self.load_mm_data(
  File "/sgl-workspace/sglang/python/sglang/srt/managers/multimodal_processors/base_processor.py", line 235, in load_mm_data
    raise RuntimeError(f"An exception occurred while loading images: {e}")
RuntimeError: An exception occurred while loading images: cannot identify image file <_io.BytesIO object at 0x7f4923fe2a70>

But once I switched to another image, it works.

[adarshxs]

hey @KCFindstr, please recheck your image URL. The error comes from PIL trying to open whatever was returned at your URL and discovering it isn’t image data at all, but likely an HTML page. SGLang's function simply streams the bytes at that URL into PIL, so if it’s not raw image bytes, PIL will bail out.

sglang/python/sglang/srt/utils.py

Lines 579 to 583 in 4c31ae9

	elif image_file.startswith("http://") or image_file.startswith("https://"):
	timeout = int(os.getenv("REQUEST_TIMEOUT", "3"))
	response = requests.get(image_file, stream=True, timeout=timeout).raw
	image = Image.open(response)
	response.close()

Make sure your URL:

• ends in the image’s extension - "png", "jpg", "jpeg", "webp", "gif".
• raw-image formats
• returns an image/* Content-Type header (you can verify with curl -I <url>).
• is publicly accessible (no auth redirect).

Example:
This will result in an error: https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png
This will work well: https://github.com/sgl-project/sglang/blob/main/test/lang/example_image.png?raw=true

[adarshxs]

Please let me know if anything else persists. Maybe we can close this now?