Repository navigation
Closed
Description
Im trying to use this on databricks inside the notebook that's running on top of a 8xA10 single node cluster, I'm initialising like:
from sglang import function, system, user, assistant, gen, set_default_backend, Runtime
runtime = Runtime("/local_disk0/mistralai/Mixtral-8x7B-Instruct-v0.1")
set_default_backend(runtime)
However I get this issue
router init state: Traceback (most recent call last):
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-51dd0ee1-a396-4939-81a6-75e3afe59af5/lib/python3.10/site-packages/sglang/srt/managers/router/manager.py", line 68, in start_router_process
model_client = ModelRpcClient(server_args, port_args)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-51dd0ee1-a396-4939-81a6-75e3afe59af5/lib/python3.10/site-packages/sglang/srt/managers/router/model_rpc.py", line 448, in __init__
self.model_server.exposed_init_model(0, server_args, port_args)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-51dd0ee1-a396-4939-81a6-75e3afe59af5/lib/python3.10/site-packages/sglang/srt/managers/router/model_rpc.py", line 54, in exposed_init_model
self.model_runner = ModelRunner(
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-51dd0ee1-a396-4939-81a6-75e3afe59af5/lib/python3.10/site-packages/sglang/srt/managers/router/model_runner.py", line 213, in __init__
torch.cuda.set_device(self.tp_rank)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-51dd0ee1-a396-4939-81a6-75e3afe59af5/lib/python3.10/site-packages/torch/cuda/__init__.py", line 404, in set_device
torch._C._cuda_setDevice(device)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-51dd0ee1-a396-4939-81a6-75e3afe59af5/lib/python3.10/site-packages/torch/cuda/__init__.py", line 284, in _lazy_init
raise RuntimeError(
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
Metadata
Metadata
Assignees
Labels
No labels
Activity
merrymercy commentedon Jan 18, 2024
Did you import torch or use any GPU-enabled libraries before creating the
Runtime?Here are some things you can try:
Runtimebefore you using any other GPU libraries.nivibilla commentedon Jan 18, 2024
still gives me the same error.
says that "context already set", if I try force = true, the cell keeps running forever with no output.
cell also runs forever.
For clarification, I am able to run the server using the web terminal on databricks and manually starting the server. However that's not ideal as I can't automate the process.
nivibilla commentedon Jan 18, 2024
Seem to have found a workaround.
A few fixes for parallel search (sgl-project#38)
[SW-228141]: Fix concurrent compilation issue (sgl-project#38)
[mllama] Modify the SDPA call to use Flash Attention (#38)
Copy OSS code from a06bf66 on 20250910 (#38)
[Feature] Implement `Streaming Mask` and add `Block Sparse Interface`. (
[BUG] Fix wrong default threshold for check discrepancy (sgl-project#38)
fix: remove .mdx extension from benchmark card links in cookbook intro (
[sgl-project#38] launcher --fp4-native-mixed (with --fp8-uniform-marl…
[sgl-project#38] bench: runs on sm_86 too (native FP4 lanes skipped),…
[sgl-project#38] projection: context-aware P model from #PGAP fits (r…
28 remaining items
Import RDNA4 patch 0019: gfx1201-add-norm-mxfp4
Import RDNA4 patch 0020: gfx1201-silu-mul-mxfp4
Import RDNA4 patch 0021: gfx1201-verify-attention-b1-tile
Import RDNA4 patch 0022: gfx1201-mxfp4-kq3
Import RDNA4 patch 0023: gfx1201-extend-attention-tile32
Import RDNA4 patch 0024: gfx1201-p1-from-kq3
Import RDNA4 patch 0025: gfx1201-prefill-custom-all-reduce
Import RDNA4 patch 0026: gfx1201-fp8-kv-dequant
Import RDNA4 patch 0027: gfx1201-fp8-kv-hw-decode
Import RDNA4 patch 0028: upstream-47057-swa-checkpoint-windows
Import RDNA4 patch 0029: upstream-47058-dflash-draft-kv-bundling
Import RDNA4 patch 0030: gfx1201-draft-kv-bundling
Import RDNA4 patch 0031: gfx1201-elastic-kv-tail-allocator
Import RDNA4 patch 0032: gfx1201-elastic-kv-tail-buffers
Import RDNA4 patch 0033: gfx1201-elastic-kv-tail-control