Repository navigation
Description
env:
2080Ti * 2
cuda_12.3.r12.3/compiler.33567101_0
python3.9
pip install "sglang[all]"
error:
new fill batch. #seq: 1. #cached_token: 0. #new_token: 8. #remaining_req: 0. #running_req: 0. tree_cache_hit_rate: 0.00%.
python: /project/lib/Analysis/Allocation.cpp:40: std::pair<llvm::SmallVector, llvm::SmallVector > mlir::triton::getCvtOrder(mlir::Attribute, mlir::Attribute): Assertion `!(srcMmaLayout && dstMmaLayout) && "Unexpected mma -> mma layout conversion"' failed.
detailed log:
(sglang2) ma@ubuntu-server:~$ python -m sglang.launch_server --model-path Qwen/Qwen1.5-0.5B --host 0.0.0.0 --port 1235 --mem-fraction-static 0.9 --tp 2
config.json: 661B [00:00, 48.4kB/s]
tokenizer_config.json: 1.16kB [00:00, 105kB/s]
vocab.json: 2.78MB [00:00, 6.09MB/s]
merges.txt: 1.67MB [00:00, 7.41MB/s]
tokenizer.json: 7.03MB [00:01, 6.60MB/s]
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
server started on [0.0.0.0]:10008
server started on [0.0.0.0]:10009
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
accepted ('127.0.0.1', 38192) with fd 30
welcome ('127.0.0.1', 38192)
accepted ('127.0.0.1', 56310) with fd 26
welcome ('127.0.0.1', 56310)
Rank 0: load weight begin.
Rank 1: load weight begin.
INFO 02-17 04:29:34 weight_utils.py:163] Using model weights format ['.safetensors']
INFO 02-17 04:29:34 weight_utils.py:163] Using model weights format ['.safetensors']
model.safetensors: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.24G/1.24G [01:52<00:00, 11.0MB/s]
Rank 1: load weight end.
Rank 0: load weight end.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Rank 0: max_total_num_token=382390, max_prefill_num_token=63731, context_len=32768, model_mode=[]
Rank 1: max_total_num_token=382390, max_prefill_num_token=63731, context_len=32768, model_mode=[]
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO: Started server process [135494]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:1235 (Press CTRL+C to quit)
INFO: 127.0.0.1:56426 - "GET /get_model_info HTTP/1.1" 200 OK
new fill batch. #seq: 1. #cached_token: 0. #new_token: 8. #remaining_req: 0. #running_req: 0. tree_cache_hit_rate: 0.00%.
python: /project/lib/Analysis/Allocation.cpp:40: std::pair<llvm::SmallVector, llvm::SmallVector > mlir::triton::getCvtOrder(mlir::Attribute, mlir::Attribute): Assertion !(srcMmaLayout && dstMmaLayout) && "Unexpected mma -> mma layout conversion"' failed. python: /project/lib/Analysis/Allocation.cpp:40: std::pair<llvm::SmallVector<unsigned int>, llvm::SmallVector<unsigned int> > mlir::triton::getCvtOrder(mlir::Attribute, mlir::Attribute): Assertion !(srcMmaLayout && dstMmaLayout) && "Unexpected mma -> mma layout conversion"' failed.
Process Process-1:
Traceback (most recent call last):
File "/home/ma/ENTER/envs/sglang2/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/ma/ENTER/envs/sglang2/lib/python3.9/multiprocessing/process.py", line 108, in run
self.target(self._args, **self._kwargs)
File "/home/ma/ENTER/envs/sglang2/lib/python3.9/site-packages/sglang/srt/managers/router/manager.py", line 79, in start_router_process
loop.run_until_complete(router.loop_for_forward())
File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
File "/home/ma/ENTER/envs/sglang2/lib/python3.9/site-packages/sglang/srt/managers/router/manager.py", line 38, in loop_for_forward
out_pyobjs = await self.model_client.step(next_step_input)
File "/home/ma/ENTER/envs/sglang2/lib/python3.9/site-packages/sglang/srt/managers/router/model_rpc.py", line 635, in _func
await asyncio.gather([asyncio.to_thread(t.wait) for t in tasks])
File "/home/ma/ENTER/envs/sglang2/lib/python3.9/asyncio/threads.py", line 25, in to_thread
return await loop.run_in_executor(None, func_call)
File "/home/ma/ENTER/envs/sglang2/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/ma/ENTER/envs/sglang2/lib/python3.9/site-packages/rpyc/core/async.py", line 51, in wait
self._conn.serve(self._ttl)
File "/home/ma/ENTER/envs/sglang2/lib/python3.9/site-packages/rpyc/core/protocol.py", line 438, in serve
data = self._channel.poll(timeout) and self._channel.recv()
File "/home/ma/ENTER/envs/sglang2/lib/python3.9/site-packages/rpyc/core/channel.py", line 55, in recv
header = self.stream.read(self.FRAME_HEADER.size)
File "/home/ma/ENTER/envs/sglang2/lib/python3.9/site-packages/rpyc/core/stream.py", line 280, in read
raise EOFError("connection closed by peer")
EOFError: connection closed by peer
HTTPConnectionPool(host='0.0.0.0', port=1235): Read timed out. (read timeout=60)
Activity
maxin9966 commentedon Feb 18, 2024
Is anyone in the same situation as me?
horiacristescu commentedon Feb 18, 2024
I am getting a similar timeout on Mistral
both the
"Unexpected mma -> mma layout conversionandRead timed out. (read timeout=60)python -m sglang.launch_server --model-path TheBloke/Mistral-7B-Merge-14-v0.1-GPTQ --port 30000I failed running the samples, it just hangs forever no message
CSWellesSun commentedon Feb 19, 2024
I encounter the same problem when I test
test/srt/model/test_llama_low_api.py. I usepython test_llama_low_api.pyand then it crashes with the same output. The error happens in thePrefillfunction.IrelandC commentedon Feb 28, 2024
I am getting a similar timeout on Qwen-7B-chat.
(GPU is NVIDIA A800. Python version is 3.8)
When I called the
/generateinterface, a timeout warning occurredThe error info 'HTTPConnectionPool(host='127.0.0.1', port=30000): Read timed out. (read timeout=60)' comes from the function
launch_serverofsglang/srt/server.py:After debugging, I found it broke at
await event.wait()ofsglang/srt/managers/tokenizer_manager.py.The function
wait()comes fromasyncio/locks.py, and debugging breaks atawait fut:IrelandC commentedon Feb 28, 2024
Has this problem been solved?
maxin9966 commentedon Mar 1, 2024
No model can load successfully, I don't know what the problem is, no one answers me
Seumi commentedon Mar 6, 2024
I've encountered the same issue, have you solved it?
Jasonsey commentedon Mar 13, 2024
I have the same issue
zhaohm14 commentedon Mar 18, 2024
I am encountering the same issue too. In my case, the problem occurs while loading the local version of llava-v1.6-34b.
stevezxzhou commentedon Mar 18, 2024
same issue
sneglen commentedon Mar 24, 2024
I had a similar issue with Mistral and a workaround was to update triton to 2.2.0 from 2.1.0. I found a hint here.
It triggers a dependency error where pip requires triton==2.1.0 with torch 2.1.2, but so far it seems to work fine.
torch 2.1.2 requires triton==2.1.0; platform_system == "Linux" and platform_machine == "x86_64", but you have triton 2.2.0 which is incompatible.My original error:
pikaro commentedon Mar 29, 2024
Can confirm that updating Triton to 2.2.0 on Linux x64 helped for me - same dependency error of course, but without any apparent negative consequences. Original error was with
TheBloke/Mistral-7B-Instruct-v0.2-GPTQ, so this seems unrelated to the model.m0g1cian commentedon Apr 14, 2024
I had the same
HTTPConnectionPool(host='127.0.0.1', port=30000): Read timed out. (read timeout=60)problem when I was using Python 3.8 on a CentOS 7 machine with 8*A800.After using a recompiled Python 3.11 with OpenSSL 1.1.1+ to install all the dependcies, there is no
Read timed outissue anymore when--tpis enabled. Simply update toTriton==2.2.0doesn't solve the issue for me and it caused KV cache leak when I was trying to start sglang server. I am currently stick withTriton==2.1.0I've also tested
--enable-flashinferoption after installingflashinfer-0.0.3+cu121torch2.1-cp311-cp311-linux_x86_64.whl, everything seems to work fine.Iven2132 commentedon May 15, 2024
Did anyone get any solution? I'm getting same error with llava-next-72b
M0gician commentedon May 15, 2024
I had the same issue when I was using Python 3.8. After switching to Python 3.11 and reinstall Sglang. It seems to work.
I'll definitely suggest people to try newer python version to see if it works
1 remaining item
github-actions commentedon Jul 25, 2024
This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.
feat: add cursor dsv3 phoenix ut (sgl-project#199)
Move and adaptation of GDN opt from Qwen3next to Qwen3.5 (sgl-project…
Port the optimizations from Qwen3-next to Qwen-3.5 (sgl-project#200)
Merge pull request sgl-project#199 from shun8686/prtest-tescases-lxl
sgl-project#194 joint_decide: wire T34 DP into the live decision path
sgl-project#199 forecast trajectory term: structure + expose decode_b…
sgl-project#199 audit closure: NaN/inf input guard + forecast union +…
sgl-project#194/sgl-project#199 holistic-review closure: shutdown Nam…
DESIGN 584 addendum 4: priced flip axis, enforcer prerequisite, joint…
docs(advice): 主 AI 回应 068/069 双核验全接受——E109 SKIP 四态 helper + direct sc…
fix(ruler): sgl-project#199 068 指针感知 SKIP 探针 + 069 direct scorer fail…