Eval bug: -fit on ignoring the VRAM required by the built-in MTP leads to OOM crash

[ali0une]

Name and Version

./bin/llama-cli --version
version: 9274 (52fb93a)
built with GNU 12.2.0 for Linux x86_64

Operating systems

Linux

GGML backends

CUDA

Hardware

12th Gen Intel(R) Core(TM) i5-12600K
NVIDIA GeForce RTX 3090

Models

Qwen3.6-27B-MTP-GGUF Q4_K_M

Problem description & steps to reproduce

-fit on only optimizing memory for the primary model context, while completely ignoring the VRAM required by the built-in MTP (Multi-Token Prediction) draft model, leading to server crash.

compile flags
cmake -B . --fresh -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="86" -DGGML_CUDA_FA_ALL_QUANTS=ON

-fit off launch command works 100%
./bin/llama-server -m /whatever/Qwen3.6-27B-MTP-Q4_K_M.gguf --parallel 1 --n-gpu-layers all --host 127.0.0.1 --port 5000 --flash-attn on --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 --cache-type-k q8_0 --cache-type-v q8_0 --kv-unified --sleep-idle-seconds 10 -fit off --ctx-size 131072 --verbose --log-verbosity 4

-fit on launch command crashes at inference time
./bin/llama-server -m /whatever/Qwen3.6-27B-MTP-Q4_K_M.gguf --parallel 1 --n-gpu-layers all --host 127.0.0.1 --port 5000 --flash-attn on --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 --cache-type-k q8_0 --cache-type-v q8_0 --kv-unified --sleep-idle-seconds 10 --fit on --fit-ctx 131072 --verbose --log-verbosity 4

Immediately after the main model loads, llama.cpp creates a second context for MTP speculative decoding. It defaults to the model's native n_ctx = 262144 (should be 131072) and tries to allocate 990.95 GB for compute buffers. Since VRAM is already exhausted by the main model, cudaMalloc fails and the server exits.

First Bad Commit

git rev-parse --short HEAD
52fb93a

Relevant log output

--fit successfully shrinks the main context:

Logs

0.25.620.084 I common_params_fit_impl: context size reduced from 262144 to 170240 -> need 3283 MiB less memory in total
0.25.620.088 I common_params_fit_impl: entire model can be fit by reducing context
0.25.620.090 I common_fit_params: successfully fit params to free device memory

MTP draft context tries to initialize with the original context length (ctx should be 131072 not 262144):

Logs

srv    load_model: creating MTP draft context against the target model...
llama_context: n_ctx         = 262144
0.27.740.184 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 990.95 MiB on device 0: cudaMalloc failed: out of memory

Then fails at inference time when sched_reserve tries to allocate the actual compute buffers for both the main and draft models, it exceeds the remaining VRAM, causing an OOM crash:

Logs

0.27.639.655 I srv  params_from_: Chat format: peg-native
0.27.639.826 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
0.27.639.828 I srv  get_availabl: updating prompt cache
0.27.639.830 I srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
0.27.639.832 I srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 170240 tokens, 8589934592 est)
0.27.639.834 I srv  get_availabl: prompt cache update took 0.01 ms
0.27.639.871 I slot launch_slot_: id  0 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> ?temp-ext -> dist 
0.27.639.878 I slot launch_slot_: id  0 | task -1 | sampler params: 
	repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
	dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 171520
	top_k = 20, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 1.000
	mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000, adaptive_target = -1.000, adaptive_decay = 0.900
0.27.639.879 I slot launch_slot_: id  0 | task 0 | processing task, is_child = 0
0.27.639.885 I slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 170240, n_keep = 0, task.n_tokens = 16
0.27.639.887 I slot update_slots: id  0 | task 0 | cached n_tokens = 0, memory_seq_rm [0, end)
0.27.711.086 I sched_reserve: reserving ...
0.27.740.184 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 990.95 MiB on device 0: cudaMalloc failed: out of memory
0.27.740.189 E ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 1039083520
0.27.740.189 E graph_reserve: failed to allocate compute buffers

Long logs:

log-fit-off.1.log

log-fit-on.1.log

[am17an]

can you check whether #23485 resolves your issue?

[ali0une]

@am17an many thanks!

Tested and if i'm not misleading i think it works only in non router mode ... the --cpu-moe argument in router mode results in crash (don't know which commit made this happen)

git rev-parse --short HEAD
c473dc0

non router mode works with or without --cpu-moe:
./bin/llama-server --log-colors off --verbose --log-verbosity 4 --log-file ~/0/log-fit-on.2.log -m /whatever/Qwen3.6-27B-MTP-Q4_K_M.gguf --parallel 1 --batch-size 8192 --ubatch-size 1024 --n-gpu-layers auto --spec-draft-ngl auto --host 127.0.0.1 --port 5000 --flash-attn on --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 --cache-type-k q8_0 --cache-type-v q8_0 --kv-unified --sleep-idle-seconds 10 --no-mmap --mlock --cpu-moe

log-fit-on.3.log

router mode without --cpu-moe works nice :
./bin/llama-server --log-colors off --log-verbosity 4 --log-file /whatever/log-fit-on-router.2.log --models-max 1 --models-preset /whatever/llama.cpp-router-config.ini --parallel 1 --batch-size 8192 --ubatch-size 1024 --n-gpu-layers auto --spec-draft-ngl auto --host 127.0.0.1 --port 5000 --flash-attn on --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 --cache-type-k q8_0 --cache-type-v q8_0 --kv-unified --sleep-idle-seconds 10 --no-mmap --mlock

Here is the config.ini:
llama.cpp-router-config.ini.txt

Relevant log output

--fit successfully shrinks the main context:

Logs

0.00.521.396 I common_params_fit_impl: getting device memory data for initial parameters:
0.00.797.653 I common_memory_breakdown_print: | memory breakdown [MiB] | total    free     self   model   context   compute    unaccounted |
0.00.797.657 I common_memory_breakdown_print: |   - CUDA0 (RTX 3090)   | 24126 = 23403 + (25610 = 15621 +    9152 +     836) +      -24888 |
0.00.797.657 I common_memory_breakdown_print: |   - Host               |                   1214 =   682 +       0 +     532                |
0.00.827.037 I common_params_fit_impl: projected to use 25610 MiB of device memory vs. 23403 MiB of free device memory
0.00.827.039 I common_params_fit_impl: cannot meet free memory target of 2404 MiB, need to reduce device memory by 4611 MiB
0.01.104.596 I common_memory_breakdown_print: | memory breakdown [MiB] | total    free     self   model   context   compute    unaccounted |
0.01.104.599 I common_memory_breakdown_print: |   - CUDA0 (RTX 3090)   | 24126 = 23403 + (20927 = 15621 +    4800 +     505) +      -20204 |
0.01.104.600 I common_memory_breakdown_print: |   - Host               |                    958 =   682 +       0 +     276                |
0.01.142.391 I common_params_fit_impl: context size reduced from 262144 to 132864 -> need 4619 MiB less memory in total
0.01.142.395 I common_params_fit_impl: entire model can be fit by reducing context
0.01.142.396 I common_fit_params: successfully fit params to free device memory
0.01.142.412 I common_fit_params: fitting params to free memory took 0.62 seconds

MTP draft context initialize with the original context length ( still don't understand why ctx is not 131072 but 262144):

Logs

0.03.406.620 I srv    load_model: creating MTP draft context against the target model '/whatever/Qwen3.6-27B-MTP-Q4_K_M.gguf'
0.03.406.672 I llama_context: constructing llama_context
0.03.406.672 I llama_context: n_seq_max     = 1
0.03.406.673 I llama_context: n_ctx         = 262144
0.03.406.673 I llama_context: n_ctx_seq     = 262144
0.03.406.673 I llama_context: n_batch       = 2048
0.03.406.673 I llama_context: n_ubatch      = 512
0.03.406.673 I llama_context: causal_attn   = 1
0.03.406.674 I llama_context: flash_attn    = enabled
0.03.406.674 I llama_context: kv_unified    = true
0.03.406.678 I llama_context: freq_base     = 10000000.0
0.03.406.679 I llama_context: freq_scale    = 1
0.03.406.679 I llama_context: n_rs_seq      = 0
0.03.406.793 I llama_context:  CUDA_Host  output buffer size =     0.95 MiB
0.03.410.003 I llama_kv_cache:      CUDA0 KV buffer size =   544.00 MiB
0.03.410.662 I llama_kv_cache: size =  544.00 MiB (262144 cells,   1 layers,  1/1 seqs), K (q8_0):  272.00 MiB, V (q8_0):  272.00 MiB
0.03.410.663 I llama_kv_cache: attn_rot_k = 1, n_embd_head_k_all = 256
0.03.410.663 I llama_kv_cache: attn_rot_v = 1, n_embd_head_k_all = 256
0.03.410.793 I sched_reserve: reserving ...
0.03.413.998 I sched_reserve: resolving fused Gated Delta Net support:
0.03.414.136 I sched_reserve: fused Gated Delta Net (autoregressive) enabled
0.03.414.186 I sched_reserve: fused Gated Delta Net (chunked) enabled
0.03.520.611 I sched_reserve:      CUDA0 compute buffer size =   836.28 MiB
0.03.520.615 I sched_reserve:  CUDA_Host compute buffer size =   532.29 MiB
0.03.520.615 I sched_reserve: graph nodes  = 63
0.03.520.615 I sched_reserve: graph splits = 2
0.03.520.616 I sched_reserve: reserve took 109.82 ms, sched copies = 1
0.03.525.328 I srv    load_model: initializing slots, n_slots = 1
0.03.544.081 I common_context_can_seq_rm: the context supports bounded partial sequence removal
0.03.579.425 I common_speculative_impl_draft_mtp: adding speculative implementation 'draft-mtp'
0.03.579.429 I common_speculative_impl_draft_mtp: - n_max=2, n_min=0, p_min=0.00, n_embd=5120, backend_sampling=1
0.03.579.431 I common_speculative_impl_draft_mtp: - gpu_layers=-1, cache_k=q8_0, cache_v=q8_0, ctx_tgt=yes, ctx_dft=yes, devices=[default]
0.03.579.746 I srv    load_model: speculative decoding context initialized
0.03.579.747 I slot   load_model: id  0 | task -1 | new slot, n_ctx = 171520

Then succeeds at inference time when sched_reserve tries to allocate the actual compute buffers for both the main and draft models:

Logs

0.10.667.229 I srv  params_from_: Chat format: peg-native
0.10.667.351 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
0.10.667.351 I srv  get_availabl: updating prompt cache
0.10.667.354 I srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
0.10.667.356 I srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 132864 tokens, 8589934592 est)
0.10.667.356 I srv  get_availabl: prompt cache update took 0.01 ms
0.10.667.399 I slot launch_slot_: id  0 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> ?temp-ext -> dist 
0.10.667.406 I slot launch_slot_: id  0 | task -1 | sampler params: 
	repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
	dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 132864
	top_k = 20, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 1.000
	mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000, adaptive_target = -1.000, adaptive_decay = 0.900
0.10.667.407 I slot launch_slot_: id  0 | task 0 | processing task, is_child = 0
0.10.667.412 I slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 132864, n_keep = 0, task.n_tokens = 16
0.10.667.414 I slot update_slots: id  0 | task 0 | cached n_tokens = 0, memory_seq_rm [0, end)
0.10.740.550 I sched_reserve: reserving ...
0.10.860.886 I sched_reserve:      CUDA0 compute buffer size =   990.95 MiB
0.10.860.889 I sched_reserve:  CUDA_Host compute buffer size =   532.29 MiB
0.10.860.890 I sched_reserve: graph nodes  = 69
0.10.860.890 I sched_reserve: graph splits = 2
0.10.860.891 I sched_reserve: reserve took 120.34 ms, sched copies = 1
0.10.865.472 I slot update_slots: id  0 | task 0 | cached n_tokens = 12, memory_seq_rm [12, end)

Long logs:

log-fit-on.2.log

router mode with --cpu-moe crashes :
./bin/llama-server --log-colors off --log-verbosity 4 --log-file /whatever/log-fit-on-cpu-moe-router.2.log --models-max 1 --models-preset /whatever/llama.cpp-router-config.ini --parallel 1 --batch-size 8192 --ubatch-size 1024 --n-gpu-layers auto --spec-draft-ngl auto --host 127.0.0.1 --port 5000 --flash-attn on --spec-draft-type-k q8_0 --spec-draft-type-v q8_0 --cache-type-k q8_0 --cache-type-v q8_0 --kv-unified --sleep-idle-seconds 10 --no-mmap --mlock --cpu-moe

Relevant log output

--fit successfully shrinks the main context:

Logs

0.00.448.381 I common_params_fit_impl: getting device memory data for initial parameters:
0.00.750.361 I common_memory_breakdown_print: | memory breakdown [MiB] | total    free     self   model   context   compute    unaccounted |
0.00.750.365 I common_memory_breakdown_print: |   - CUDA0 (RTX 3090)   | 24126 = 23403 + (26446 = 15621 +    9152 +    1672) +      -25724 |
0.00.750.365 I common_memory_breakdown_print: |   - Host               |                   1746 =   682 +       0 +    1064                |
0.00.788.429 I common_params_fit_impl: projected to use 26446 MiB of device memory vs. 23403 MiB of free device memory
0.00.788.432 I common_params_fit_impl: cannot meet free memory target of 3240 MiB, need to reduce device memory by 6283 MiB
0.01.067.460 I common_memory_breakdown_print: | memory breakdown [MiB] | total    free     self   model   context   compute    unaccounted |
0.01.067.466 I common_memory_breakdown_print: |   - CUDA0 (RTX 3090)   | 24126 = 23403 + (21432 = 15621 +    4800 +    1010) +      -20709 |
0.01.067.466 I common_memory_breakdown_print: |   - Host               |                   1234 =   682 +       0 +     552                |
0.01.106.933 I common_params_fit_impl: context size reduced from 262144 to 131072 -> need 5014 MiB less memory in total

MTP draft context initialize with the original context length ( still don't understand why ctx is not 131072 but 262144):

Logs

0.03.654.086 I srv    load_model: creating MTP draft context against the target model '/whatever/Qwen3.6-27B-MTP-Q4_K_M.gguf'
0.03.654.134 I llama_context: constructing llama_context
0.03.654.134 I llama_context: n_seq_max     = 1
0.03.654.135 I llama_context: n_ctx         = 262144
0.03.654.135 I llama_context: n_ctx_seq     = 262144
0.03.654.135 I llama_context: n_batch       = 8192
0.03.654.135 I llama_context: n_ubatch      = 1024
0.03.654.135 I llama_context: causal_attn   = 1
0.03.654.136 I llama_context: flash_attn    = enabled
0.03.654.136 I llama_context: kv_unified    = true
0.03.654.138 I llama_context: freq_base     = 10000000.0
0.03.654.139 I llama_context: freq_scale    = 1
0.03.654.139 I llama_context: n_rs_seq      = 0
0.03.654.231 I llama_context:  CUDA_Host  output buffer size =     0.95 MiB
0.03.656.949 I llama_kv_cache:      CUDA0 KV buffer size =   544.00 MiB
0.03.657.607 I llama_kv_cache: size =  544.00 MiB (262144 cells,   1 layers,  1/1 seqs), K (q8_0):  272.00 MiB, V (q8_0):  272.00 MiB
0.03.657.608 I llama_kv_cache: attn_rot_k = 1, n_embd_head_k_all = 256
0.03.657.608 I llama_kv_cache: attn_rot_v = 1, n_embd_head_k_all = 256
0.03.657.736 I sched_reserve: reserving ...
0.03.661.567 I sched_reserve: resolving fused Gated Delta Net support:
0.03.661.696 I sched_reserve: fused Gated Delta Net (autoregressive) enabled
0.03.661.771 I sched_reserve: fused Gated Delta Net (chunked) enabled
0.03.831.218 I sched_reserve:      CUDA0 compute buffer size =  1672.30 MiB
0.03.831.221 I sched_reserve:  CUDA_Host compute buffer size =  1064.30 MiB
0.03.831.221 I sched_reserve: graph nodes  = 63
0.03.831.221 I sched_reserve: graph splits = 2
0.03.831.222 I sched_reserve: reserve took 173.49 ms, sched copies = 1
0.03.835.625 W srv    load_model: cache_reuse is not supported by this context, it will be disabled
0.03.835.630 I srv    load_model: initializing slots, n_slots = 1
0.03.850.920 I common_context_can_seq_rm: the context supports bounded partial sequence removal
0.03.886.676 I common_speculative_impl_draft_mtp: adding speculative implementation 'draft-mtp'
0.03.886.679 I common_speculative_impl_draft_mtp: - n_max=2, n_min=0, p_min=0.00, n_embd=5120, backend_sampling=1
0.03.886.680 I common_speculative_impl_draft_mtp: - gpu_layers=-1, cache_k=q8_0, cache_v=q8_0, ctx_tgt=yes, ctx_dft=yes, devices=[default]
0.03.887.069 I srv    load_model: speculative decoding context initialized
0.03.887.070 I slot   load_model: id  0 | task -1 | new slot, n_ctx = 131072

Then fails at inference time when sched_reserve tries to allocate the actual compute buffers for both the main and draft models:

Logs

[52157] 0.03.915.256 I srv  params_from_: Chat format: peg-native
[52157] 0.03.915.475 I slot get_availabl: id  0 | task -1 | selected slot by LRU, t_last = -1
[52157] 0.03.915.477 I srv  get_availabl: updating prompt cache
[52157] 0.03.915.479 I srv          load:  - looking for better prompt, base f_keep = -1.000, sim = 0.000
[52157] 0.03.915.482 I srv        update:  - cache state: 0 prompts, 0.000 MiB (limits: 8192.000 MiB, 131072 tokens, 8589934592 est)
[52157] 0.03.915.483 I srv  get_availabl: prompt cache update took 0.01 ms
[52157] 0.03.915.510 I reasoning-budget: activated, budget=2048 tokens
[52157] 0.03.915.524 I slot launch_slot_: id  0 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> ?min-p -> ?xtc -> temp-ext -> dist 
[52157] 0.03.915.531 I slot launch_slot_: id  0 | task -1 | sampler params: 
[52157] 	repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
[52157] 	dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 131072
[52157] 	top_k = 20, top_p = 0.950, min_p = 0.000, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.600
[52157] 	mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000, adaptive_target = -1.000, adaptive_decay = 0.900
[52157] 0.03.915.532 I slot launch_slot_: id  0 | task 0 | processing task, is_child = 0
[52157] 0.03.915.553 I slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 131072, n_keep = 0, task.n_tokens = 21
[52157] 0.03.915.555 I slot update_slots: id  0 | task 0 | cached n_tokens = 0, memory_seq_rm [0, end)
[52157] 0.04.011.243 I sched_reserve: reserving ...
[52157] 0.04.071.133 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 1980.95 MiB on device 0: cudaMalloc failed: out of memory
[52157] 0.04.071.137 E ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 2077173760
[52157] 0.04.071.138 E graph_reserve: failed to allocate compute buffers
[52157] /whatever/llama.cpp/bin/libggml-base.so.0(+0x18ba8)[0x7f7ebdd5eba8]
[52157] /whatever/llama.cpp/bin/libggml-base.so.0(ggml_print_backtrace+0x1e4)[0x7f7ebdd5ef84]
[52157] /whatever/llama.cpp/bin/libggml-base.so.0(+0x2b299)[0x7f7ebdd71299]
[52157] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xa8e1a)[0x7f7ebd6a8e1a]
[52157] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xa8e85)[0x7f7ebd6a8e85]
[52157] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xa90d8)[0x7f7ebd6a90d8]
[52157] /whatever/llama.cpp/bin/libllama.so.0(+0x92882)[0x7f7ebda92882]
[52157] /whatever/llama.cpp/bin/libllama.so.0(_ZN13llama_context6decodeERK11llama_batch+0x234)[0x7f7ebdad3b54]
[52157] /whatever/llama.cpp/bin/libllama.so.0(llama_decode+0xb)[0x7f7ebdad545b]
[52157] /whatever/llama.cpp/bin/libllama-common.so.0(_ZN33common_speculative_impl_draft_mtp7processERK11llama_batch+0x2a5)[0x7f7ebe0b3945]
[52157] /whatever/llama.cpp/bin/libllama-common.so.0(_Z26common_speculative_processP18common_speculativeRK11llama_batch+0x30)[0x7f7ebe0ab670]
[52157] /whatever/llama.cpp/bin/llama-server(+0x10f545)[0x55688c4c2545]
[52157] /whatever/llama.cpp/bin/llama-server(+0x198011)[0x55688c54b011]
[52157] /whatever/llama.cpp/bin/llama-server(+0x74c4d)[0x55688c427c4d]
[52157] /lib/x86_64-linux-gnu/libc.so.6(+0x2724a)[0x7f7ebd44524a]
[52157] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85)[0x7f7ebd445305]
[52157] /whatever/llama.cpp/bin/llama-server(+0x6e061)[0x55688c421061]
[52157] terminate called after throwing an instance of 'std::runtime_error'
[52157]   what():  failed to allocate compute pp buffers
0.09.859.823 E srv    operator(): http client error: Failed to read connection
0.09.859.875 I srv  log_server_r: done request: POST /v1/chat/completions 127.0.0.1 500
0.46.201.264 I srv    operator(): operator(): cleaning up before exit...
0.46.201.278 I srv    unload_all: stopping model instance name=Qwen3.6-27B-MTP-Q4_K_M
0.46.201.756 I srv    operator(): instance name=Qwen3.6-27B-MTP-Q4_K_M exited with status 1

Long logs:

log-fit-on-cpu-moe-router.2.log

[am17an]

It's a bit tough to cover all scenarios like this. Since router mode is experimental I would treat this as a separate issue for now

[ali0une]

It's a bit tough to cover all scenarios like this. Since router mode is experimental I would treat this as a separate issue for now

Allright. Many thanks @am17an for the PR fixing this issue.

Should i wait for your PR to be merged to open a new issue about the --cpu-moe with my latest answer?