Repository navigation
Conversation
…MASK When running multiple Intel XPU GPUs in multiprocessing mode, worker processes would concurrently initialize all Level Zero devices, causing a crash in libur_adapter_level_zero_v2.so during device initialization. Changes: - multiproc_executor.py: Temporarily set ZE_AFFINITY_MASK per-worker before spawning to isolate each worker to its own device - xpu_worker.py: Use device index 0 when ZE_AFFINITY_MASK restricts visibility to a single device (local_rank >= device_count) - core.py: Enable ZE_AFFINITY_MASK setup for XPU in Ray DP path (previously a no-op) - ray_executor.py: Add ZE_AFFINITY_MASK to worker-specific env vars Co-authored-by: GitHub Copilot Co-authored-by: hongbolv <33214277+hongbolv@users.noreply.github.com>
- Move original_ze_mask inside XPU platform check for better scoping - Use device_count == 1 condition for clearer intent in device index selection Co-authored-by: GitHub Copilot Co-authored-by: hongbolv <33214277+hongbolv@users.noreply.github.com>
hongbolv
left a comment
hongbolv
left a comment
There was a problem hiding this comment.
The cause is not correct. When setting the tensor_parallel_size=1, the crash is still there.
hongbolv
left a comment
hongbolv
left a comment
There was a problem hiding this comment.
@copilot Your root cause is not correct. With new command line with tensor_parallel_size=1, the crash is still there. The example is:# SPDX-License-Identifier: Apache-2.0
Demo: Qwen1.5-MoE-A2.7B on 2x Intel Arc Pro B60 with TP=2
import os
Triton is not available on XPU; disable torch.compile to avoid errors
os.environ.setdefault("TORCH_COMPILE_DISABLE", "1")
XPU model loading can be slow; increase the engine startup timeout
os.environ.setdefault("VLLM_ENGINE_READY_TIMEOUT_S", "1800")
from vllm import LLM, SamplingParams
def main():
model_path = "/home/media/Hongbo/Qwen1.5-MoE-A2.7B"
print("=" * 60)
print("Qwen1.5-MoE-A2.7B Demo with Tensor Parallelism = 1")
print("Device: Intel Arc Pro B60 (XPU)")
print("=" * 60)
# Initialize the model with TP=2
# enforce_eager=True since we are not using Triton
# num_gpu_blocks_override limits KV cache to avoid OOM on GPU 1
llm = LLM(
model=model_path,
tensor_parallel_size=1,
trust_remote_code=True,
dtype="float16",
max_model_len=256,
enforce_eager=True,
gpu_memory_utilization=0.95,
num_gpu_blocks_override=100,
)
sampling_params = SamplingParams(
temperature=0.7,
top_p=0.9,
max_tokens=128,
)
prompts = [
"Hello, my name is",
"The capital of France is",
"Explain quantum computing in simple terms:",
"Write a short poem about artificial intelligence:",
]
print("\nGenerating responses...\n")
outputs = llm.generate(prompts, sampling_params)
print("=" * 60)
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
print(f"Prompt: {prompt!r}")
print(f"Response: {generated_text!r}")
print("-" * 60)
print("\nDone!")
if name == "main":
main()
…ch.xpu access The crash occurs even with tensor_parallel_size=1 because Level Zero tries to initialize all devices on the system when torch.xpu is first accessed. Set ZE_AFFINITY_MASK in XPUWorker.init_device() before any torch.xpu call to restrict Level Zero to a single device per worker. Co-authored-by: GitHub Copilot Co-authored-by: hongbolv <33214277+hongbolv@users.noreply.github.com>
You're right — the root cause is that Level Zero tries to initialize ALL devices on the system when any Fixed in db1e07d: |
There was a problem hiding this comment.
@copilot (vllm) media@media-NF5468M7-4U-emr:/Hongbo/networking.dataplane.fabricx.vllm$ nm -D /home/media/miniforge3/envs/vllm/lib/libur_adapter_level_zero_v2.so.0 | grep urDeviceGetInfo/Hongbo/networking.dataplane.fabricx.vllm$
(vllm) media@media-NF5468M7-4U-emr:
urDeviceGetInfo 没有找到吗？
hongbolv
left a comment
hongbolv
left a comment
There was a problem hiding this comment.
@copilot 从crash core dump来看，怎么会有指针访问出现？解释一下原因
Thread 1 "python3" received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()
(gdb) bt
#0 0x0000000000000000 in ?? ()
#1 0x00007ffe192f89be in std::_Function_handler<void (ZeStruct<ze_device_vector_width_properties_ext_t>&), ur_device_handle_t::initialize(int, int)::$_11>::_M_invoke(std::_Any_data const&, ZeStruct<_ze_device_vector_width_properties_ext_t>&) ()
from /home/media/miniforge3/envs/vllm/lib/libur_adapter_level_zero_v2.so.0
#2 0x00007ffff7ca7e6b in __pthread_once_slow (once_control=0x5555651d1778,
init_routine=0x7fffb06edf30 std::__once_proxy()) at ./nptl/pthread_once.c:116
#3 0x00007ffff7ca7ed9 in _pthread_once (once_control=, init_routine=)
at ./nptl/pthread_once.c:143
#4 0x00007ffe192f03b1 in ur::level_zero::urDeviceGetInfo(ur_device_handle_t, ur_device_info_t, unsigned long, void, unsigned long*) ()
from /home/media/miniforge3/envs/vllm/lib/libur_adapter_level_zero_v2.so.0
#5 0x00007fff8f3d24ef in ur_tracing_layer::urDeviceGetInfo(ur_device_handle_t, ur_device_info_t, unsigned long, void, unsigned long*) () from /home/media/miniforge3/envs/vllm/lib/libur_loader.so.0
#6 0x00007fff8f32237d in urDeviceGetInfo ()
from /home/media/miniforge3/envs/vllm/lib/libur_loader.so.0
#7 0x00007fffb0d35bc1 in decltype(auto) sycl::_V1::detail::device_impl::get_info_abi_workaround<sycl::_V1::info::device::half_fp_config, false>() const ()
from /home/media/miniforge3/envs/vllm/lib/python3.12/site-packages/torch/lib/../../../../libsycl.so.8
#8 0x00007fffb0d30efa in sycl::_V1::detail::ABINeutralT<sycl::_V1::detail::is_device_info_descsycl::_V1::info::device::half_fp_config::return_type>::type sycl::_V1::device::get_info_implsycl::_V1::info::device::half_fp_config() const ()
from /home/media/miniforge3/envs/vllm/lib/python3.12/site-packages/torch/lib/../../../../libsycl.so.8
#9 0x00007ffff6e844b2 in c10::xpu::(anonymous namespace)::initDeviceProperties(c10::xpu::DeviceProp*, signed char) ()
from /home/media/miniforge3/envs/vllm/lib/python3.12/site-packages/torch/lib/libc10_xpu.so
#10 0x00007fffb3cb59af in at::xpu::getDeviceProperties(signed char) ()
The
The backtrace shows: Here's what's happening:
Why |
|
@copilot It doesn't make sense. All the devices are same ARC device. So all the device handle should populate same extension function. There is no case which you mentioned"When multiple devices are visible, the UR adapter initializes all device handles. If one device's driver doesn't properly populate extension function pointers, the null dereference happens during that device's pthread_once initialization. " |
You're correct — my previous explanation was wrong. Since all devices are identical Arc Pro B60s, they use the same driver and would have the same extension support. The null function pointer isn't about one device being different from another. The more accurate explanation: the crash is a bug in the UR Level Zero adapter's multi-device initialization path. When multiple identical devices are visible, Setting |
Co-authored-by: GitHub Copilot Co-authored-by: hongbolv <33214277+hongbolv@users.noreply.github.com>
|
@copilot 不要用work around，需要找到root cause去解决这个问题 |
|
@copilot (vllm) media@media-NF5468M7-4U-emr: |
# 找到提供 libze_loader 的包
dpkg -S /lib/x86_64-linux-gnu/libze_loader.so.1.20.3
# 或搜索可用的 level-zero 相关包
apt list 2>/dev/null | grep -i "level-zero\|libze"常见的包名变体有 sudo apt install --only-upgrade <实际包名>如果 PPA 中没有更新版本，可以直接从 GitHub 下载最新的 # 从 https://github.com/oneapi-src/level-zero/releases 下载
wget https://github.com/oneapi-src/level-zero/releases/download/v1.20.2/level-zero_1.20.2+u22.04_amd64.deb
sudo dpkg -i level-zero_1.20.2+u22.04_amd64.deb注意选择与你 Ubuntu 版本匹配的 |
|
@copilot (vllm) media@media-NF5468M7-4U-emr:~$ apt list 2>/dev/null | grep -i "level-zero|libze" |
包名确认了： Ubuntu plucky 仓库中 # 下载 1.21+ 版本（需要包含 zeDeviceGetVectorWidthPropertiesExt）
# 从 https://github.com/oneapi-src/level-zero/releases 找到适合的版本
wget https://github.com/oneapi-src/level-zero/releases/download/v1.21.1/level-zero_1.21.1+u22.04_amd64.deb
sudo dpkg -i level-zero_1.21.1+u22.04_amd64.deb安装后验证： import ctypes
ze = ctypes.CDLL("libze_loader.so.1")
fn = ze.zeDeviceGetVectorWidthPropertiesExt
print(f"Found: {fn}") # 应该输出函数地址而非报错如果 git clone https://github.com/oneapi-src/level-zero.git
cd level-zero && git checkout v1.21.1
mkdir build && cd build
cmake .. && make -j$(nproc)
sudo make install
关键发现： Ubuntu plucky 仓库中 方法1：从 GitHub releases 直接安装 # 下载 Level Zero 1.21+ 版本（需要包含 zeDeviceGetVectorWidthPropertiesExt）
# 访问 https://github.com/oneapi-src/level-zero/releases 找到适合的版本
# 例如：
wget https://github.com/oneapi-src/level-zero/releases/download/v1.21.9/level-zero_1.21.9+u24.04_amd64.deb
sudo dpkg -i level-zero_1.21.9+u24.04_amd64.deb方法2：添加 Intel APT 源获取更新的包 wget -qO - https://repositories.intel.com/gpu/intel-graphics.key | sudo gpg --dearmor -o /usr/share/keyrings/intel-graphics.gpg
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.gpg] https://repositories.intel.com/gpu/ubuntu noble unified" | sudo tee /etc/apt/sources.list.d/intel-gpu.list
sudo apt update
apt list -a libze1 2>/dev/null # 查看可用版本
sudo apt install libze1升级后验证： ls -la /lib/x86_64-linux-gnu/libze_loader.so.1
python3 -c "import ctypes; ze=ctypes.CDLL('libze_loader.so.1'); print(ze.zeDeviceGetVectorWidthPropertiesExt)"如果第二条命令不再报 |
|
@copilot libze-intel-gpu1 版本 26.05.37020.3-0 对应的loader版本是多少？ |
建议直接升级到参考平台使用的 1.24.1 版本。可以从 GitHub releases 下载： # 从 https://github.com/oneapi-src/level-zero/releases 找到 v1.24.1
# 下载 libze_loader 和 libze_intel_gpu 的 deb 包
# 或者从源码编译：
git clone https://github.com/oneapi-src/level-zero.git
cd level-zero && git checkout v1.24.1
mkdir build && cd build
cmake .. && make -j$(nproc)
sudo make install安装后用之前的 |
所以从 https://github.com/oneapi-src/level-zero/releases 下载的
https://github.com/oneapi-src/level-zero/releases 包含的是 Level Zero loader（ Level Zero 的两个组件分别在不同的仓库：
你的问题是 loader 太旧（1.20.3），需要从 level-zero releases 下载更新的 loader 升级到 1.21+ (建议直接到参考平台的 1.24.1)。 |
|
@copilot (vllm) media@media-NF5468M7-4U-emr:~/Hongbo/level-zero/build$ su |
编译安装的 1.24.1 在 方法1：更新系统符号链接（推荐） sudo ln -sf /usr/local/lib/libze_loader.so.1.24.1 /usr/lib/x86_64-linux-gnu/libze_loader.so.1
sudo ldconfig方法2：用 export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH加到 方法3：用 ldconfig 注册 echo "/usr/local/lib" | sudo tee /etc/ld.so.conf.d/local.conf
sudo ldconfig安装后验证： # 确认链接到新版本
ldconfig -p | grep libze_loader
# 应该显示 /usr/local/lib/libze_loader.so.1.24.1
# 确认函数可用
python3 -c "import ctypes; ze=ctypes.CDLL('libze_loader.so.1'); print(ze.zeDeviceGetVectorWidthPropertiesExt)"方法3最干净，因为
新编译的 1.24.1 已经安装到 方法1：设置 export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
python debug.py # 验证是否能找到 zeDeviceGetVectorWidthPropertiesExt方法2：更新 ldconfig（永久生效） # 确保 /usr/local/lib 在 ldconfig 搜索路径中
echo "/usr/local/lib" | sudo tee /etc/ld.so.conf.d/local-level-zero.conf
sudo ldconfig
# 验证链接是否指向新版本
ldconfig -p | grep libze_loader方法3：直接替换符号链接 sudo ln -sf /usr/local/lib/libze_loader.so.1.24.1 /usr/lib/x86_64-linux-gnu/libze_loader.so.1
sudo ldconfig建议先用方法1测试，确认 crash 消失后再用方法2永久生效。验证命令： python -c "import ctypes; ze=ctypes.CDLL('libze_loader.so.1'); print(ze.zeDeviceGetVectorWidthPropertiesExt)"如果输出函数地址而非报错，说明新 loader 已生效。 |
|
@copilot (vllm) media@media-NF5468M7-4U-emr:~/Hongbo/level-zero/build$ ldconfig -p | grep libze_loader |
验证是否生效： # 确认实际加载的是新版本
python3 -c "import ctypes; ze=ctypes.CDLL('libze_loader.so.1'); print(ze.zeDeviceGetVectorWidthPropertiesExt)"如果输出函数地址（如 如果仍然报 sudo ldconfig
# 然后重新验证验证通过后，可以直接运行 vllm 测试是否不再 crash。
# 确认实际链接的版本
ls -la /usr/local/lib/libze_loader.so.1
# 应该指向 libze_loader.so.1.24.1
# 确认函数可用
python3 -c "import ctypes; ze=ctypes.CDLL('libze_loader.so.1'); print(ze.zeDeviceGetVectorWidthPropertiesExt)"
# 应该输出函数地址，不报错
# 最终测试：不设 ZE_AFFINITY_MASK，直接触发多设备初始化
python3 -c "import torch; print(torch.xpu.device_count())"
# 如果输出 4 而不是 SEGFAULT，说明 loader 升级成功如果
echo "/usr/local/lib" | sudo tee /etc/ld.so.conf.d/local-level-zero.conf
sudo ldconfig然后验证： ldconfig -p | grep libze_loader
# /usr/local/lib/libze_loader.so.1 应该排在第一位
python3 debug.py
# 应该输出 Found: 而非 NOT found如果 |
|
@copilot (vllm) media@media-NF5468M7-4U-emr:~/Hongbo/networking.dataplane.fabricx.vllm$ python3 -c "import ctypes; ze=ctypes.CDLL('libze_loader.so.1'); print(ze.zeDeviceGetVectorWidthPropertiesExt)" |
新 loader 已生效 — # 不设 ZE_AFFINITY_MASK，触发多设备初始化
python3 -c "import torch; print(torch.xpu.device_count())"
# 应该输出 4 而不是 SEGFAULT如果输出 4，说明系统层面的 loader/driver 版本不匹配问题已解决。我们的 vllm 改动（设置
python3 -c "import ctypes; ze=ctypes.CDLL('libze_loader.so.1'); print(ze.zeDeviceGetVectorWidthPropertiesExt)"如果输出
python3 -c "import ctypes; ze=ctypes.CDLL('libze_loader.so.1'); print(ze.zeDeviceGetVectorWidthPropertiesExt)"如果输出
新 loader 1.24.1 已生效， python3 -c "import torch; print(torch.xpu.device_count())"如果输出 我们的 vllm 改动（设置 |
On systems with multiple Intel XPU GPUs (e.g., Intel Arc), the UR Level Zero adapter crashes in
libur_adapter_level_zero_v2.soduringur_device_handle_t_::initialize— a null function pointer dereference atoneapi-src/unified-runtimedevice.cpp:2065. The UR adapter checksUrPlatform->zeDriverExtensionMapforZE_DEVICE_VECTOR_SIZES_EXT_NAME(present), then callszeDeviceGetVectorWidthPropertiesExtthrough the Level Zero DDI table (ze_device_dditable_t.pfnGetVectorWidthPropertiesExt), but the DDI table function pointer isnullptr— a version mismatch between the Level Zero loader and driver where the driver reports extension support but the loader doesn't populate the function pointer.ZE_CALL_NOCHECKhas no null guard, so it calls0x0→ SEGFAULT. This occurs even withtensor_parallel_size=1because the firsttorch.xpucall in the worker triggers Level Zero to enumerate and initialize all visible devices.The upstream fix should be in
oneapi-src/unified-runtime: add a null check before callingzeDeviceGetVectorWidthPropertiesExt, or usezeDriverGetExtensionFunctionAddressto load the function pointer explicitly (as the adapter already does for other extensions at line 2204).At the vllm level, the fix sets
ZE_AFFINITY_MASK(XPU's equivalent ofCUDA_VISIBLE_DEVICES) to isolate each worker to its assigned physical device before anytorch.xpuaccess. This is the standard XPU device isolation mechanism — the same pattern vllm already uses withCUDA_VISIBLE_DEVICESfor CUDA — and also avoids the upstream UR bug by reducing visible devices to 1.Changes
xpu_worker.py(key fix): SetZE_AFFINITY_MASKat the start ofinit_device()before anytorch.xpucall, mappinglocal_rankto a physical device viadevice_id_to_physical_device_id(). After setting the mask, only one device is visible so device index 0 is always used. This handles both TP=1 (UniProcExecutor) and TP>1 (MultiprocExecutor) paths.multiproc_executor.py: Additionally setZE_AFFINITY_MASKper-worker beforeproc.start()so the child process inherits device isolation from the start. Restored infinallyblock.core.py: Remove XPU-specificpassin_set_visible_devices()— was preventingZE_AFFINITY_MASKfrom being set in the Ray DP engine core actor path.ray_executor.py: AddZE_AFFINITY_MASKtoWORKER_SPECIFIC_ENV_VARSso it isn't propagated from driver to all workers.💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.