Repository navigation
[Fix] scheduler/multimodal: Fix embedding over-length hang, cap CPU pool, fix ST6 reference - #48372
dayanandav wants to merge 2 commits into
Conversation
This comment has been minimized.
This comment has been minimized.
|
@siju-samuel @ANSHUMAN87 help to review. |
|
[by Claude Code] Measurements in support of this change. They show that the LLaVA warmup timeout comes from starting the CPU preprocess pool, not from the image work itself. Context. Setup.
1. CPython starts every worker on the first 2. First-submit cost of a fresh fork pool. These runs used
3. End to end, the server's warmup image request. This is the step bounded by the 10 s
Caveat: I don't know the core count of the |
…pool, fix ST6 reference Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
3c06e78 to
4187028
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Motivation
Three backend-independent bugs found while fixing test timeouts:
os.cpu_count()fork workers, all started on the first request. On a 512-core host that request overruns its timeout.test_embedding_modelsis wrong: plain text goes through the chat template, and the non-ST branch is not causal (the kwarg was renamed).Modifications
scheduler.py:set_finish_with_abortwhen embedding length validation fails, as the generate path does.base_processor.py: default pool sizemin(os.cpu_count(), 16).SGLANG_CPU_WORKERSstill overrides it. Changes the default only on hosts with more than 16 cores.test/runners.py: on sentence-transformers >= 6, drop the chat-template modality and passis_causalunder the new kwarg name. No-op below v6.test_embedding_length_validation.py(regression for 1) andtest_base_processor_cpu_pool.py(CPU unit test for 2).Accuracy Tests
No model-output change. Tested on Intel Arc Pro B60 (no CUDA run yet):
test_embedding_length_validation: passes; without the fix it fails withReadTimeout.test_base_processor_cpu_pool: passes; fails without the fix.test_embedding_models(test_prefill_logits,test_embedding_encoding_format, sentence-transformers 6.0.1): pass.test_vision_openai_server_a(video_images, multi_images): pass.Speed Tests and Profiling
N/A: an error path, a pool-size default and test code.
🤖 Generated with Claude Code
CI States
Latest PR Test (Base): ❌ Run #43064852556
Latest PR Test (Extra): ❌ Run #43064852646
Latest PR Test (AMD ROCm 10): ❌ Run #43064852455
Tip
Base CI only covers the core cases, and its workload may not exercise your change. Use
/rerun-testto run the tests your change touches, or copy this prompt for an agent: