Skip to content

[Fix/Potential Bugs] Can not correctly import models in python/sglang/srt/models - #311

Merged
merrymercy merged 1 commit into
sgl-project:mainfrom
Luodian:fix_import_models
Mar 22, 2024
Merged

merrymercy merged 1 commit into
sgl-project:mainfrom
Luodian:fix_import_models

Conversation

@Luodian

@Luodian Luodian commented Mar 19, 2024

Copy link
Copy Markdown
Contributor

Associated with issue #310

@Qubitium

Qubitium commented Mar 19, 2024 •

Copy link
Copy Markdown
Contributor
# prep env
pip uninstall sglang # make sure my env has no sglang
git clone https://github.com/sgl-project/sglang
cd sglang 
pip install ./python
cd examples/quick_start

excecute example directly via python is ok.

python srt_example_llava.py <--  no error

running with debuggy as in your stacktrace and got error but different than your stacktrace:

pyton -m debugpy --listen localhost:5678  srt_example_llava.py <-- error
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/root/miniconda3/lib/python3.11/site-packages/debugpy/__main__.py", line 39, in <module>
    cli.main()
  File "/root/miniconda3/lib/python3.11/site-packages/debugpy/server/cli.py", line 430, in main
    run()
  File "/root/miniconda3/lib/python3.11/site-packages/debugpy/server/cli.py", line 284, in run_file
    runpy.run_path(target, run_name="__main__")
  File "/root/miniconda3/lib/python3.11/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
    return _run_module_code(code, init_globals, run_name,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/lib/python3.11/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/root/miniconda3/lib/python3.11/site-packages/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "srt_example_llava.py", line 61, in <module>
    stream()
  File "srt_example_llava.py", line 28, in stream
    for out in state.text_iter("answer"):
  File "/root/miniconda3/lib/python3.11/site-packages/sglang/lang/interpreter.py", line 675, in text_iter
    event = self.stream_executor.stream_var_event[var_name]
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
KeyError: 'answer'

Is debugpy doing something strange with paths?

@comaniac comaniac left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM. Is it possible to add a unit test to make sure we are at the good shape?

@BabyChouSr

Copy link
Copy Markdown
Contributor

I had a similar issue trying to run the LLaVA model, and this works for me! Thanks for the good work :)

@merrymercy
merrymercy merged commit 08df63a into sgl-project:main Mar 22, 2024
@merrymercy

Copy link
Copy Markdown
Collaborator

@comaniac @BabyChouSr Adding some unit tests will be ideal. We have some tests but they require GPUs to run. I am not sure how to make them work with GitHub CI.
https://github.com/sgl-project/sglang/blob/main/docs/test_process.md#srt-unit-tests

timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
vschandramourya pushed a commit to vschandramourya/sglang that referenced this pull request Feb 3, 2026
efschu added a commit to efschu/htsglang that referenced this pull request Jul 30, 2026
…22.0% vs vLLM sgl-project#707), code 108.25 (+21.8%), fair-vs-voll within joint spread => the +22-25% lead is essentially ALL engine/placement/transport at equal k, adaptive-k share small-to-undetectable; sgl-project#311 FALSIFIED with reversed sign — fp8 draft is SLOWER (+4 to +12% ms/Verify vs bf16; bs=1 drafter is latency- not bandwidth-bound, per-step dequant adds cost; accept holds), bs=1 verdict hardens to -34 to -50%; side proofs: draft-quant flag verified not to reach target, extra reserve requirement belongs to high-accept ladder not AEON; harness gap: battery_release_locks is process-local, orphans locks from new shells
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants