Skip to content

LlaVa Usage with server option ValueError: ... not in list #41

Closed

Description

@aliozts

Hello, I tried to utilize sglang backend with LlaVa model utilizing the command that's provided in the README

I created a new environment and installed sglang with pip install "sglang[all]".

python3 -m sglang.launch_server --model-path liuhaotian/llava-v1.5-7b --tokenizer-path llava-hf/llava-1.5-7b-hf --port 30000
import requests
import json
# includes an image of a cat
path = "images/cat_2.jpeg"
text = "what is this?"
# checked https://github.com/sgl-project/sglang/blob/main/python/sglang/srt/managers/io_struct.py#L9 for input
data = {"text": text,
        "image_data": path}

headers = {'Content-Type': 'application/json'}

response = requests.post('http://localhost:30000/generate', json=data, headers=headers)

print(response.json())

it gets stuck in runtime(jupyter, does not raise the error) but in server I see the error

ValueError: 32000 is not in list

May I ask if there is something I'm doing wrong?

Activity

  1. merrymercy commented on Jan 18, 2024

    @merrymercy
    Collaborator

    Yeah, your prompt format is slightly wrong.

    Try this

    import requests
    import json
    
    path = "test/lang/image.png"
    
    data = {
        "text": "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: <image>\nWhat is this? ASSISTANT:",
        "image_data": path
    }
    
    headers = {'Content-Type': 'application/json'}
    
    response = requests.post('http://localhost:30000/generate', json=data, headers=headers)
    
    print(response.json())

    Instead of using the low-level API, you can also try our frontend language.
    See examples for batch processing and single query

  2. aliozts commented on Jan 18, 2024

    @aliozts
    Author

    Hi, thank you for prompt response! just an update about this.

    Exception in ModelRpcClient:
    Traceback (most recent call last):
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/managers/router/model_rpc.py", line 130, in exposed_step
        self.forward_step()
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/managers/router/model_rpc.py", line 145, in forward_step
        self.forward_fill_batch(new_batch)
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/managers/router/model_rpc.py", line 328, in forward_fill_batch
        logits, normalized_logprobs = self.model_runner.forward(
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/managers/router/model_runner.py", line 436, in forward
        return self.forward_extend_multi_modal(**kwargs)
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/managers/router/model_runner.py", line 413, in forward_extend_multi_modal
        return self.model.forward(
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/models/llava.py", line 122, in forward
        return self.language_model(
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/models/llama2.py", line 269, in forward
        hidden_states = self.model(input_ids, positions, input_metadata, skip_embed)
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/models/llama2.py", line 239, in forward
        hidden_states, residual = layer(
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/models/llama2.py", line 191, in forward
        hidden_states = self.self_attn(
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/models/llama2.py", line 140, in forward
        attn_output = self.attn(q, k, v, input_metadata)
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
        return self._call_impl(*args, **kwargs)
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
        return forward_call(*args, **kwargs)
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/layers/radix_attention.py", line 140, in forward
        return self.extend_forward(q, k, v, input_metadata)
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/layers/radix_attention.py", line 65, in extend_forward_triton
        extend_attention_fwd(
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/sglang/srt/layers/extend_attention.py", line 240, in extend_attention_fwd
        _fwd_kernel[grid](
      File "<string>", line 63, in _fwd_kernel
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/triton/compiler/compiler.py", line 425, in compile
        so_path = make_stub(name, signature, constants)
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/triton/compiler/make_launcher.py", line 39, in make_stub
        so = _build(name, src_path, tmpdir)
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/triton/common/build.py", line 61, in _build
        cuda_lib_dirs = libcuda_dirs()
      File "/opt/conda/envs/sglang/lib/python3.10/site-packages/triton/common/build.py", line 21, in libcuda_dirs
        libs = subprocess.check_output(["ldconfig", "-p"]).decode()
      File "/opt/conda/envs/sglang/lib/python3.10/subprocess.py", line 421, in check_output
        return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
      File "/opt/conda/envs/sglang/lib/python3.10/subprocess.py", line 503, in run
        with Popen(*popenargs, **kwargs) as process:
      File "/opt/conda/envs/sglang/lib/python3.10/subprocess.py", line 971, in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
      File "/opt/conda/envs/sglang/lib/python3.10/subprocess.py", line 1863, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'ldconfig'

    I'm getting the above error in GCP L4 instance when I try with single GPU or 2 GPUs with(--tp 2). I will check the error myself too but if it helps anyone to look at if they're using this too.

  3. Ying1123 commented on Jan 18, 2024

    @Ying1123
    Contributor

    @aliozts I have a note for solving this issue:

    Solution 1: If the command is not found by ldconfig but found by sudo ldconfig, then copy /usr/sbin/ldconfig to /usr/bin/ldconfig and sudo chmod -R a+w /opt/conda/lib.
    Solution 2: export PATH=$PATH:/sbin/.

  4. aliozts commented on Jan 18, 2024

    @aliozts
    Author

    thank you so much for your help, solution 1 worked for me!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions