Skip to content

How to use inside notebook? #38

Closed
Closed

Description

@nivibilla

Im trying to use this on databricks inside the notebook that's running on top of a 8xA10 single node cluster, I'm initialising like:

from sglang import function, system, user, assistant, gen, set_default_backend, Runtime
runtime = Runtime("/local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1")
set_default_backend(runtime)

However I get this issue

router init state: Traceback (most recent call last):
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-51dd0ee1-a396-4939-81a6-75e3afe59af5/lib/python3.10/site-packages/sglang/srt/managers/router/manager.py", line 68, in start_router_process
    model_client = ModelRpcClient(server_args, port_args)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-51dd0ee1-a396-4939-81a6-75e3afe59af5/lib/python3.10/site-packages/sglang/srt/managers/router/model_rpc.py", line 448, in __init__
    self.model_server.exposed_init_model(0, server_args, port_args)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-51dd0ee1-a396-4939-81a6-75e3afe59af5/lib/python3.10/site-packages/sglang/srt/managers/router/model_rpc.py", line 54, in exposed_init_model
    self.model_runner = ModelRunner(
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-51dd0ee1-a396-4939-81a6-75e3afe59af5/lib/python3.10/site-packages/sglang/srt/managers/router/model_runner.py", line 213, in __init__
    torch.cuda.set_device(self.tp_rank)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-51dd0ee1-a396-4939-81a6-75e3afe59af5/lib/python3.10/site-packages/torch/cuda/__init__.py", line 404, in set_device
    torch._C._cuda_setDevice(device)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-51dd0ee1-a396-4939-81a6-75e3afe59af5/lib/python3.10/site-packages/torch/cuda/__init__.py", line 284, in _lazy_init
    raise RuntimeError(
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method

Activity

  1. merrymercy commented on Jan 18, 2024

    @merrymercy
    Collaborator

    Did you import torch or use any GPU-enabled libraries before creating the Runtime?

    Here are some things you can try:

    1. create Runtime before you using any other GPU libraries.
    2. try this at the very beginning of your notebook.
    import multiprocessing as mp
    mp.set_start_method('spawn')
    1. see this colab notebook example https://colab.research.google.com/drive/13lOJt8uFYZJetqQIudAlK8oJJX8PENNk?usp=sharing
  2. nivibilla commented on Jan 18, 2024

    @nivibilla
    ContributorAuthor
    1. still gives me the same error.

    2. says that "context already set", if I try force = true, the cell keeps running forever with no output.

    3. cell also runs forever.

    For clarification, I am able to run the server using the web terminal on databricks and manually starting the server. However that's not ideal as I can't automate the process.

  3. nivibilla commented on Jan 18, 2024

    @nivibilla
    ContributorAuthor

    Seem to have found a workaround.

    import subprocess
    server = subprocess.Popen("python -m sglang.launch_server --model-path /local_disk0/mistralai/Mistral-7B-Instruct-v0.2/ --port 30000 --tp 8", shell=True)
    
  4. added a commit that references this issue on Sep 10, 2025
  5. added 3 commits that reference this issue on Sep 25, 2026
  6. 28 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions