Skip to content

Import Errors occurring even when dependencies are installed #403

Closed

Description

@david-vectorflow

I am unable to create a Runtime with sglang as follows runtime = sgl.Runtime(model_path=MODEL_DIR, tokenizer_path=MODEL_DIR). It throws the error below:

ImportError: cannot import name '_set_default_torch_dtype' from 'vllm.model_executor.model_loader' 
(/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/__init__.py)

I am running this on an A100 with the Nvidia PyTorch base image recommended by the Flash Attention repo ("nvcr.io/nvidia/pytorch:24.03-py3"). I have confirmed via print statements that all the dependencies are installing in the desired location and the files are actually there. I have tried installing sglang both with pip and from source and get the exact same error.

I noticed that not every package in the file structure has an __init__.py file, which can cause issues. Perhaps the version of PyTorch in the base image conflicts somehow with what sglang expects? Aside from that, I am not sure why this is happening. But given how simple this code is, I suspect there is an issue with the library, not the machine.

Activity

  1. nikita-lukashevich1 commented on May 1, 2024

    @nikita-lukashevich1

    Look here: #391

  2. david-vectorflow commented on May 1, 2024

    @david-vectorflow
    Author

    Thanks! Looks like it still isn't solved though?

  3. github-actions commented on Jul 25, 2024

    @github-actions
    Contributor

    This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.

  4. added a commit that references this issue on Apr 21, 2026
  5. added 2 commits that reference this issue on Aug 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions