Skip to content

[Fix] scheduler/multimodal: Fix embedding over-length hang, cap CPU pool, fix ST6 reference - #48372

Open
dayanandav wants to merge 2 commits into
sgl-project:mainfrom
dayanandav:fix-embedding-hang-mm-pool-st6
Open

dayanandav wants to merge 2 commits into
sgl-project:mainfrom
dayanandav:fix-embedding-hang-mm-pool-st6

Conversation

@dayanandav

@dayanandav dayanandav commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Three backend-independent bugs found while fixing test timeouts:

  1. An embedding prompt that fits the context length but not the KV pool fails validation but is never finished, so the request hangs until the client times out.
  2. The multimodal CPU preprocess pool defaults to os.cpu_count() fork workers, all started on the first request. On a 512-core host that request overruns its timeout.
  3. With sentence-transformers 6, the HF reference in test_embedding_models is wrong: plain text goes through the chat template, and the non-ST branch is not causal (the kwarg was renamed).

Modifications

  • scheduler.py: set_finish_with_abort when embedding length validation fails, as the generate path does.
  • base_processor.py: default pool size min(os.cpu_count(), 16). SGLANG_CPU_WORKERS still overrides it. Changes the default only on hosts with more than 16 cores.
  • test/runners.py: on sentence-transformers >= 6, drop the chat-template modality and pass is_causal under the new kwarg name. No-op below v6.
  • New tests: test_embedding_length_validation.py (regression for 1) and test_base_processor_cpu_pool.py (CPU unit test for 2).

Accuracy Tests

No model-output change. Tested on Intel Arc Pro B60 (no CUDA run yet):

  • test_embedding_length_validation: passes; without the fix it fails with ReadTimeout.
  • test_base_processor_cpu_pool: passes; fails without the fix.
  • test_embedding_models (test_prefill_logits, test_embedding_encoding_format, sentence-transformers 6.0.1): pass.
  • LLaVA test_vision_openai_server_a (video_images, multi_images): pass.

Speed Tests and Profiling

N/A: an error path, a pool-size default and test code.

🤖 Generated with Claude Code


CI States

Latest PR Test (Base): ❌ Run #43064852556
Latest PR Test (Extra): ❌ Run #43064852646
Latest PR Test (AMD ROCm 10): ❌ Run #43064852455

Tip

Base CI only covers the core cases, and its workload may not exercise your change. Use /rerun-test to run the tests your change touches, or copy this prompt for an agent:

Find the registered tests this PR affects and run them with /rerun-test, following https://github.com/sgl-project/sglang/blob/main/.agents/skills/rerun-test/SKILL.md

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@github-actions

This comment has been minimized.

@dayanandav

Copy link
Copy Markdown
Contributor Author

@siju-samuel @ANSHUMAN87 help to review.

@nvpohanh

nvpohanh commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

[by Claude Code] Measurements in support of this change. They show that the LLaVA warmup timeout comes from starting the CPU preprocess pool, not from the image work itself.

Context. TestLlavaServer (test_vision_openai_server_a.py) and test_vision_chunked_prefill.py fail on h100-computable-* CI runners. The server logs Replaced a broken multimodal CPU preprocess pool exactly 10 s after its own startup warmup image request, then returns HTTP 500 and fails to initialize. Examples: job 126989270916 and job 126928003215.

Setup.

  • Code: current main (f3532057e1), model lmms-lab/llava-onevision-qwen2-0.5b-ov.
  • Host: AMD EPYC 9554 with os.cpu_count() = 256; the container's CPU affinity was 32.
  • Runtime: CPython 3.12.16, 1x L40S. The pool step runs on CPU only.

1. CPython starts every worker on the first submit(). With a fork context, ProcessPoolExecutor._safe_to_dynamically_spawn_children is False, so _start_executor_manager_thread() calls _launch_processes(). That starts all max_workers processes inside the first submit().

2. First-submit cost of a fresh fork pool. These runs used _forget_parent_executor_wakeups as the initializer, in a process that had already imported llava, torch and transformers (1.2 GB RSS). Each value is the mean of 3 runs, and the second submit always took under 1 ms.

max_workers first submit().result()
256 (os.cpu_count()) 4.22 s
128 2.07 s
64 1.02 s
16 0.25 s
4 0.07 s

3. End to end, the server's warmup image request. This is the step bounded by the 10 s REQUEST_TIMEOUT in llava.py::_process_single_image.

Config Warmup request Pool step (blocked in submit / total) Later requests
default (256 workers) 8.9 s 7.2 s / 7.8 s ~0.64 s
SGLANG_CPU_WORKERS=16 (this PR's default) 1.8 s 0.44 s / 1.04 s ~0.64 s
384 workers (emulates a bigger host) fails: pool replaced at 10.0 s, warmup HTTP 500, init failure 11.4 s blocked in submit n/a

Caveat: I don't know the core count of the h100-computable runners, because the job logs don't print it.

…pool, fix ST6 reference

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dayanandav
dayanandav force-pushed the fix-embedding-hang-mm-pool-st6 branch from 3c06e78 to 4187028 Compare November 11, 2026 17:06
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

This comment has been minimized.

@dayanandav dayanandav changed the title [Fix] scheduler, multimodal: Fix embedding over-length hang, cap CPU pool, fix ST6 reference [Fix] scheduler/multimodal: Fix embedding over-length hang, cap CPU pool, fix ST6 reference Nov 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants