You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
LlaVa Usage with server option ValueError: ... not in list #41
importrequestsimportjson# includes an image of a catpath="images/cat_2.jpeg"text="what is this?"# checked https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/managers/io_struct.py#L9 for inputdata= {"text": text,
"image_data": path}
headers= {'Content-Type': 'application/json'}
response=requests.post('http://localhost:30000/generate', json=data, headers=headers)
print(response.json())
it gets stuck in runtime(jupyter, does not raise the error) but in server I see the error
importrequestsimportjsonpath="test/lang/image.png"data= {
"text": "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image>\nWhat is this? ASSISTANT:",
"image_data": path
}
headers= {'Content-Type': 'application/json'}
response=requests.post('http://localhost:30000/generate', json=data, headers=headers)
print(response.json())
Instead of using the low-level API, you can also try our frontend language.
See examples for batch processing and single query
Hi, thank you for prompt response! just an update about this.
Exception in ModelRpcClient:
Traceback (most recent call last):
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/managers/router/model_rpc.py", line 130, in exposed_step
self.forward_step()
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/managers/router/model_rpc.py", line 145, in forward_step
self.forward_fill_batch(new_batch)
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/managers/router/model_rpc.py", line 328, in forward_fill_batch
logits, normalized_logprobs = self.model_runner.forward(
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/managers/router/model_runner.py", line 436, in forward
return self.forward_extend_multi_modal(**kwargs)
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/managers/router/model_runner.py", line 413, in forward_extend_multi_modal
return self.model.forward(
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/models/llava.py", line 122, in forward
return self.language_model(
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/models/llama2.py", line 269, in forward
hidden_states = self.model(input_ids, positions, input_metadata, skip_embed)
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/models/llama2.py", line 239, in forward
hidden_states, residual = layer(
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/models/llama2.py", line 191, in forward
hidden_states = self.self_attn(
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/models/llama2.py", line 140, in forward
attn_output = self.attn(q, k, v, input_metadata)
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/layers/radix_attention.py", line 140, in forward
return self.extend_forward(q, k, v, input_metadata)
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/layers/radix_attention.py", line 65, in extend_forward_triton
extend_attention_fwd(
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/layers/extend_attention.py", line 240, in extend_attention_fwd
_fwd_kernel[grid](
File "<string>", line 63, in _fwd_kernel
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/triton/compiler/compiler.py", line 425, in compile
so_path = make_stub(name, signature, constants)
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/triton/compiler/make_launcher.py", line 39, in make_stub
so = _build(name, src_path, tmpdir)
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/triton/common/build.py", line 61, in _build
cuda_lib_dirs = libcuda_dirs()
File "/opt/conda/envs/sglang/lib/python3.10/site-packages/triton/common/build.py", line 21, in libcuda_dirs
libs = subprocess.check_output(["ldconfig", "-p"]).decode()
File "/opt/conda/envs/sglang/lib/python3.10/subprocess.py", line 421, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/opt/conda/envs/sglang/lib/python3.10/subprocess.py", line 503, in run
with Popen(*popenargs, **kwargs) as process:
File "/opt/conda/envs/sglang/lib/python3.10/subprocess.py", line 971, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/opt/conda/envs/sglang/lib/python3.10/subprocess.py", line 1863, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'ldconfig'
I'm getting the above error in GCP L4 instance when I try with single GPU or 2 GPUs with(--tp 2). I will check the error myself too but if it helps anyone to look at if they're using this too.
Solution 1: If the command is not found by ldconfig but found by sudo ldconfig, then copy /usr/sbin/ldconfig to /usr/bin/ldconfig and sudo chmod -R a+w /opt/conda/lib.
Solution 2: export PATH=$PATH:/sbin/.
Activity
merrymercy commentedon Jan 18, 2024
Yeah, your prompt format is slightly wrong.
Try this
Instead of using the low-level API, you can also try our frontend language.
See examples for batch processing and single query
aliozts commentedon Jan 18, 2024
Hi, thank you for prompt response! just an update about this.
I'm getting the above error in GCP L4 instance when I try with single GPU or 2 GPUs with(--tp 2). I will check the error myself too but if it helps anyone to look at if they're using this too.
Ying1123 commentedon Jan 18, 2024
@aliozts I have a note for solving this issue:
Solution 1: If the command is not found by
ldconfigbut found bysudo ldconfig, then copy/usr/sbin/ldconfigto/usr/bin/ldconfigandsudo chmod -R a+w /opt/conda/lib.Solution 2:
export PATH=$PATH:/sbin/.aliozts commentedon Jan 18, 2024
thank you so much for your help, solution 1 worked for me!!
Use MParam class to eliminate __getattr__ (sgl-project#41)
[SW-228981]: set cache_size limit to high value for recompilation (sg…
Add opt fused grouped topk bias kernel (sgl-project#41)
docs: add description of `output_latency` metric in the docs (sgl-pro…
feat: add missing cookbook models (MOVA, S2-Pro) and SGLang Omni sect…
feat(dt): explicit cuda-ipc proposal admission credit (default 64)
feat(dt): explicit cuda-ipc proposal admission credit (default 64)