V0.3版本复现DeepSeek-R1-GGUF-BF16出错：Illegal instruction (core dumped)

[CodingQinghao]

【部署环境】
系统：
Ubuntu22.04
CPU:
Intel(R) Xeon(R) Platinum 8490H 支持AMX
内存：
1.5TB DDR5
显卡：
4* A800 80GB

版本：
Python3.11
cuda_12.6.0_560.28.03_linux.run
ktransformers-0.3.0rc0+cu126torch26fancy-cp311-cp311-linux_x86_64.whl
flash_attn-2.7.4.post1+cu12torch2.6cxx11abiFALSE-cp311-cp311-linux_x86_64.whl

命令：
python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path /data/data/DeepSeek-R1-GGUF-BF16/ --prompt_file /root/hello.txt --cpu_infer 120 --max_new_tokens 1000

运行日志：
(dsr1) root@ailabnode2:~# python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-R1 --gguf_path /data/data/model/ --prompt_file /root/hello.txt
using custom modeling_xxx.py.
using default_optimize_rule for DeepseekV3ForCausalLM
AVX512F
Injecting model as ktransformers.operators.models . KDeepseekV2Model
Injecting model.embed_tokens as default
Injecting model.layers as default
Injecting model.layers.0 as default
Injecting model.layers.0.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.0.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.q_a_layernorm as default
Injecting model.layers.0.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.kv_a_layernorm as default
Injecting model.layers.0.self_attn.kv_b_proj as default
Injecting model.layers.0.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbeddingV3
Injecting model.layers.0.mlp as default
Injecting model.layers.0.mlp.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.0.mlp.act_fn as default
Injecting model.layers.0.input_layernorm as default
Injecting model.layers.0.post_attention_layernorm as default
省去中间步骤
Injecting model.layers.60 as default
Injecting model.layers.60.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.60.self_attn.q_a_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.self_attn.q_a_layernorm as default
Injecting model.layers.60.self_attn.q_b_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.self_attn.kv_a_proj_with_mqa as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.self_attn.kv_a_layernorm as default
Injecting model.layers.60.self_attn.kv_b_proj as default
Injecting model.layers.60.self_attn.o_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.self_attn.rotary_emb as ktransformers.operators.RoPE . RotaryEmbeddingV3
Injecting model.layers.60.mlp as ktransformers.operators.experts . KDeepseekV3MoE
Injecting model.layers.60.mlp.experts as ktransformers.operators.experts . KTransformersExperts
Injecting model.layers.60.mlp.gate as ktransformers.operators.gate . KMoEGate
Injecting model.layers.60.mlp.shared_experts as default
Injecting model.layers.60.mlp.shared_experts.gate_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.mlp.shared_experts.up_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.mlp.shared_experts.down_proj as ktransformers.operators.linear . KTransformersLinear
Injecting model.layers.60.mlp.shared_experts.act_fn as default
Injecting model.layers.60.input_layernorm as default
Injecting model.layers.60.post_attention_layernorm as default
Injecting model.norm as default
Injecting lm_head as default
loading token_embd.weight to cpu
/root/miniconda3/envs/dsr1/lib/python3.11/site-packages/ktransformers/util/custom_gguf.py:302: UserWarning: The given NumPy array is not writable, and PyTorchy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (T
values = torch.from_numpy(values)
loading blk.0.attn_q_a_norm.weight to cuda:0
loading blk.0.attn_kv_a_norm.weight to cuda:0
loading blk.0.attn_kv_b.weight to cuda:0
loading blk.0.attn_norm.weight to cuda:0
loading blk.0.ffn_norm.weight to cuda:0
省区中间步骤
loading blk.60.attn_q_a_norm.weight to cuda:1
loading blk.60.attn_kv_a_norm.weight to cuda:1
loading blk.60.attn_kv_b.weight to cuda:1
loading blk.60.attn_norm.weight to cuda:1
loading blk.60.ffn_norm.weight to cuda:1
loading output_norm.weight to cuda:1
loading output.weight to cuda:1
Illegal instruction (core dumped)

错误：
Illegal instruction (core dumped)

[CodingQinghao]

模型来自：
https://modelscope.cn/models/unsloth/DeepSeek-R1-GGUF/files
下面的DeepSeek-R1-BF16

[CodingQinghao]

(dsr1) root@ailabnode2:~# cpuid -1 -l 7 -s 0 | grep -i amx
AMX-BF16: tile bfloat16 support = true
AMX-TILE: tile architecture support = true
AMX-INT8: tile 8-bit integer support = true

[KMSorSMS]

@chenht2022

[yuliao0214]

The same error. The log indicates the use of AVX512, even though my CPU supports the AMX instruction set. I tried out Deepseek R1 and V3, but nothing's changed.

using custom modeling_xxx.py.
using default_optimize_rule for DeepseekV3ForCausalLM
__AVX512F__
...
loading blk.59.attn_kv_a_norm.weight to cuda:1
loading blk.59.attn_kv_b.weight to cuda:1
loading blk.59.attn_norm.weight to cuda:1
loading blk.59.ffn_norm.weight to cuda:1
loading blk.60.attn_q_a_norm.weight to cuda:1
loading blk.60.attn_kv_a_norm.weight to cuda:1
loading blk.60.attn_kv_b.weight to cuda:1
loading blk.60.attn_norm.weight to cuda:1
loading blk.60.ffn_norm.weight to cuda:1
loading output_norm.weight to cuda:1
loading output.weight to cuda:1
./run.sh: line 6:   994 Illegal instruction     (core dumped) python -m ktransformers.local_chat --model_path /data00/models/DeepSeek-V3 --gguf_path /data00/models/DeepSeek-V3-BF16-gguf --prompt_file prompt.file --cpu_infer 178 --max_new_tokens 1000

Category	Details
Hardware	- CPU: Intel(R) Xeon(R) Platinum 8457C * 2 (supports amx_int8 amx_bf16) - Memory: 708GB DRAM - GPU: 4090D * 8
Operating System	Ubuntu 22.04
GPU Driver Version	560.35.05
CUDA Version	12.6
Library Versions	- ktransformers: 0.3.0rc0+cu126torch26fancy - flash_attn: 2.7.4.post1

[yuliao0214]

The AMX has not been granted permission to run. KTrans needs to utilize ARCH_REQ_XCOMP_PERM to request the necessary permission and set XFEATURE_XTILEDATA flag. As described in the kernel documentation:

ARCH_REQ_XCOMP_PERM allows to request permission for a dynamically enabled
feature or a feature set. A feature set can be mapped to a facility, e.g.
AMX, and can require one or more XSTATE components to be enabled.

For more detailed information, please refer to: https://www.kernel.org/doc/Documentation/x86/xstate.rst.
@Azure-Tang @KMSorSMS @chenht2022

Currently, we can fix this issue by preloading a dynamic-link library to request permission.

gcc -shared -fPIC -o preload_amx.so preload_amx.c
LD_PRELOAD=./preload_amx.so python -m ktransformers.local_chat --model_path $MODEL_PATH --gguf_path $GGUF_PATH --prompt_file prompt.file --cpu_infer 178 --max_new_tokens 1000

//preload_amx.c
#include <stdio.h>
#include <sys/syscall.h>
#include <unistd.h>

#define ARCH_REQ_XCOMP_PERM     0x1023
#define XFEATURE_XTILEDATA      18

__attribute__((constructor)) void init() {
  if (syscall(SYS_arch_prctl, ARCH_REQ_XCOMP_PERM, XFEATURE_XTILEDATA))
  {
     printf("\n Fail to do XFEATURE_XTILEDATA \n\n");
  }
  else
  {
     printf("\n TILE DATA USE SET - OK \n\n");
  }
}

[Atream]

Thank you, we will fix it.

[CodingQinghao]

The AMX has not been granted permission to run. KTrans needs to utilize ARCH_REQ_XCOMP_PERM to request the necessary permission and set XFEATURE_XTILEDATA flag. As described in the kernel documentation:

ARCH_REQ_XCOMP_PERM allows to request permission for a dynamically enabled
feature or a feature set. A feature set can be mapped to a facility, e.g.
AMX, and can require one or more XSTATE components to be enabled.

For more detailed information, please refer to: https://www.kernel.org/doc/Documentation/x86/xstate.rst. @Azure-Tang @KMSorSMS @chenht2022

Currently, we can fix this issue by preloading a dynamic-link library to request permission.

gcc -shared -fPIC -o preload_amx.so preload_amx.c
LD_PRELOAD=./preload_amx.so python -m ktransformers.local_chat --model_path $MODEL_PATH --gguf_path $GGUF_PATH --prompt_file prompt.file --cpu_infer 178 --max_new_tokens 1000

//preload_amx.c
#include <stdio.h>
#include <sys/syscall.h>
#include <unistd.h>

#define ARCH_REQ_XCOMP_PERM 0x1023
#define XFEATURE_XTILEDATA 18

attribute((constructor)) void init() {
if (syscall(SYS_arch_prctl, ARCH_REQ_XCOMP_PERM, XFEATURE_XTILEDATA))
{
printf("\n Fail to do XFEATURE_XTILEDATA \n\n");
}
else
{
printf("\n TILE DATA USE SET - OK \n\n");
}
}

working for me

[yansiyu550]

The AMX has not been granted permission to run. KTrans needs to utilize ARCH_REQ_XCOMP_PERM to request the necessary permission and set XFEATURE_XTILEDATA flag. As described in the kernel documentation:

ARCH_REQ_XCOMP_PERM allows to request permission for a dynamically enabled
feature or a feature set. A feature set can be mapped to a facility, e.g.
AMX, and can require one or more XSTATE components to be enabled.

For more detailed information, please refer to: https://www.kernel.org/doc/Documentation/x86/xstate.rst. @Azure-Tang @KMSorSMS @chenht2022

Currently, we can fix this issue by preloading a dynamic-link library to request permission.

gcc -shared -fPIC -o preload_amx.so preload_amx.c
LD_PRELOAD=./preload_amx.so python -m ktransformers.local_chat --model_path $MODEL_PATH --gguf_path $GGUF_PATH --prompt_file prompt.file --cpu_infer 178 --max_new_tokens 1000

//preload_amx.c
#include <stdio.h>
#include <sys/syscall.h>
#include <unistd.h>

#define ARCH_REQ_XCOMP_PERM 0x1023
#define XFEATURE_XTILEDATA 18

attribute((constructor)) void init() {
if (syscall(SYS_arch_prctl, ARCH_REQ_XCOMP_PERM, XFEATURE_XTILEDATA))
{
printf("\n Fail to do XFEATURE_XTILEDATA \n\n");
}
else
{
printf("\n TILE DATA USE SET - OK \n\n");
}
}

It works!