Skip to content

[CI] test: Give TestLlavaServer's llava preprocessing more time - #48619

Merged
nvpohanh merged 1 commit into
sgl-project:mainfrom
nvpohanh:claude/ci-fix-llava-request-timeout
Nov 6, 2026
Merged

nvpohanh merged 1 commit into
sgl-project:mainfrom
nvpohanh:claude/ci-fix-llava-request-timeout

Conversation

@nvpohanh

@nvpohanh nvpohanh commented Nov 5, 2026 •

Copy link
Copy Markdown
Collaborator

[by Claude Code]

Motivation

test/registered/vlm/test_vision_openai_server_a.py::TestLlavaServer (lmms-lab/llava-onevision-qwen2-0.5b-ov) fails in setUpClass on 1-gpu-large runners from the h100-computable-* pool. This is the same failure that #48443 fixed for test_vision_chunked_prefill.py:

[...] Replaced a broken multimodal CPU preprocess pool
File ".../multimodal/processors/llava.py", in _process_single_image
    process_future = await asyncio.wait_for(
TimeoutError
Initialization failed. warmup error: ... AssertionError: {"object":"error","message":"Internal server error: ", ... "code":500}
[Errno 98] error while attempting to bind on address ('127.0.0.1', ...): address already in use
TimeoutError: Server failed to start within the timeout period

Seen in two consecutive attempts of the same run on #47493: job 126709198594 (h100-computable-e-gpu-2) and job 126989270916 (h100-computable-d-gpu-7).

In both jobs, and in the scheduled main failure of test_vision_chunked_prefill.py (job 126928003215), the timeout fires on the server's own startup warmup image request. It comes about 10 s after Uvicorn starts and before any test request. Other VLM servers in the same partition and on the same runner (Gemma-3, InternVL, Kimi-VL, LFM2.5) start fine, because only llava's preprocessor bounds the CPU pool call with REQUEST_TIMEOUT.

The test now runs in extra-a (#47619), so it no longer blocks PR Test Base, but it still fails in PR Test Extra on those runners.

Modifications

  • python/sglang/test/vlm_utils.py: TestOpenAIMLLMServerBase gets an extra_env class attribute (default {}), passed to popen_launch_server(env=...). popen_launch_server merges it into os.environ, so the default leaves every existing test unchanged.
  • test/registered/vlm/test_vision_openai_server_a.py: TestLlavaServer sets extra_env = {"REQUEST_TIMEOUT": "60"}. This is the same value [CI] test: Give test_vision_chunked_prefill's llava preprocessing more time #48443 uses.

Test-only change; no runtime code changes. The cause is the fork-context CPU preprocess pool, which starts all os.cpu_count() workers inside its first submit(); see the measurements below and in this comment on #48372. #48372 caps that pool at 16 workers, which fixes the cause for every llava test; if it lands, this override can be dropped.

Accuracy Tests

N/A (test harness only). I checked locally with popen_launch_server stubbed: TestLlavaServer.setUpClass now passes env={"REQUEST_TIMEOUT": "60"}, and TestLfm2VlServer still passes env={}. I have not run the GPU test here; it needs a 1-gpu-large CI runner.

Speed Tests and Profiling

No runtime change. Root-cause measurements, on main f3532057e1, lmms-lab/llava-onevision-qwen2-0.5b-ov, an AMD EPYC 9554 host with os.cpu_count() = 256, and CPython 3.12.16:

Config Server warmup image request Pool step inside the 10 s REQUEST_TIMEOUT
default (256 workers) 8.9 s 7.8 s (7.2 s blocked starting workers in submit)
SGLANG_CPU_WORKERS=16 1.8 s 1.0 s
384 workers (emulates a bigger host) fails: pool replaced at 10.0 s, HTTP 500, server init fails (the CI signature) 11.4 s blocked in submit

Once the pool is up, per-image preprocessing takes about 0.57 s. Forking costs about 28-30 ms per worker. With this PR, the first request on a large host still takes about 8-11 s, but it fits within the 60 s timeout.

Checklist

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ❌ Run #42001182193
Latest PR Test (Extra): ❌ Run #42001182269
Latest PR Test (AMD ROCm 10): ❌ Run #42001182153

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

TestLlavaServer (test_vision_openai_server_a.py) fails on h100-computable-* 1-gpu-large runners: the server's startup warmup image request hits llava's 10 s REQUEST_TIMEOUT in _process_single_image, warmup returns 500, and the relaunch fails with EADDRINUSE. Launch its server with REQUEST_TIMEOUT=60, as sgl-project#48443 did for test_vision_chunked_prefill, through a new extra_env hook on TestOpenAIMLLMServerBase.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions github-actions Bot added the Multi-modal multi-modal language model label Nov 5, 2026
@nvpohanh
nvpohanh merged commit 0e8fca5 into sgl-project:main Nov 6, 2026
110 of 128 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Multi-modal multi-modal language model

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant