Eval bug: Gibberish with long context prompts GPT OSS

[thad0ctor]

Name and Version

Is anyone else getting gibberish with long context prompts?

I have tried 3 different ggufs, all the same results. Smaller prompts work fine, when I drop in a 40K context prompt and get a response of "GGGGGGGGGGGGGG.....". -fa makes no difference.

llama-server -m gpt-oss-120b-F16.gguf --threads 24 --threads-batch 48 --batch-size 4096 --ubatch-size 4096 --ctx-size 131072 --temp 0.6 --min-p 0.01 --tensor-split 20,20,20,20 --n-gpu-layers 80 --main-gpu 1 --flash-attn --no-mmap --mlock

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: yes
ggml_cuda_init: found 4 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes
  Device 1: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes
  Device 2: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes
  Device 3: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes
load_backend: loaded CUDA backend from /home/rgilbreth/Desktop/llama.cpp/llama.cpp/build/bin/libggml-cuda.so
load_backend: loaded RPC backend from /home/rgilbreth/Desktop/llama.cpp/llama.cpp/build/bin/libggml-rpc.so
load_backend: loaded CPU backend from /home/rgilbreth/Desktop/llama.cpp/llama.cpp/build/bin/libggml-cpu-icelake.so
build: 6100 (9fc3b690) with gcc-13 (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu
system info: n_threads = 24, n_threads_batch = 48, total_threads = 48

system_info: n_threads = 24 (n_threads_batch = 48) / 48 | CUDA : ARCHS = 1200 | FORCE_CUBLAS = 1 | F16 = 1 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 512 | FA_ALL_QUANTS = 1 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 

main: binding port with default address family
main: HTTP server is listening, hostname: 127.0.0.1, port: 5001, http threads: 47
main: loading model
srv    load_model: loading model '/media/T9/Models/gpt-oss-120b-mxfp4-00001-of-00003.gguf'
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 5090) - 31085 MiB free
llama_model_load_from_file_impl: using device CUDA1 (NVIDIA GeForce RTX 5090) - 31061 MiB free
llama_model_load_from_file_impl: using device CUDA2 (NVIDIA GeForce RTX 5090) - 31588 MiB free
llama_model_load_from_file_impl: using device CUDA3 (NVIDIA GeForce RTX 5090) - 31588 MiB free
llama_model_loader: additional 2 GGUFs metadata loaded.
llama_model_loader: loaded meta data with 35 key-value pairs and 687 tensors from /media/rgilbreth/T9/Models/gpt-oss-120b-mxfp4-00001-of-00003.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = gpt-oss
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Prerelease 100 120b Hf
llama_model_loader: - kv   3:                           general.finetune str              = hf
llama_model_loader: - kv   4:                           general.basename str              = prerelease-100
llama_model_loader: - kv   5:                         general.size_label str              = 120B
llama_model_loader: - kv   6:                        gpt-oss.block_count u32              = 36
llama_model_loader: - kv   7:                     gpt-oss.context_length u32              = 131072
llama_model_loader: - kv   8:                   gpt-oss.embedding_length u32              = 2880
llama_model_loader: - kv   9:                gpt-oss.feed_forward_length u32              = 2880
llama_model_loader: - kv  10:               gpt-oss.attention.head_count u32              = 64
llama_model_loader: - kv  11:            gpt-oss.attention.head_count_kv u32              = 8
llama_model_loader: - kv  12:                     gpt-oss.rope.freq_base f32              = 150000.000000
llama_model_loader: - kv  13:   gpt-oss.attention.layer_norm_rms_epsilon f32              = 0.000010
llama_model_loader: - kv  14:                       gpt-oss.expert_count u32              = 128
llama_model_loader: - kv  15:                  gpt-oss.expert_used_count u32              = 4
llama_model_loader: - kv  16:               gpt-oss.attention.key_length u32              = 64
llama_model_loader: - kv  17:             gpt-oss.attention.value_length u32              = 64
llama_model_loader: - kv  18:           gpt-oss.attention.sliding_window u32              = 128
llama_model_loader: - kv  19:         gpt-oss.expert_feed_forward_length u32              = 2880
llama_model_loader: - kv  20:                  gpt-oss.rope.scaling.type str              = yarn
llama_model_loader: - kv  21:                gpt-oss.rope.scaling.factor f32              = 32.000000
llama_model_loader: - kv  22: gpt-oss.rope.scaling.original_context_length u32              = 4096
llama_model_loader: - kv  23:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  24:                         tokenizer.ggml.pre str              = gpt-4o
llama_model_loader: - kv  25:                      tokenizer.ggml.tokens arr[str,201088]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  26:                  tokenizer.ggml.token_type arr[i32,201088]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  27:                      tokenizer.ggml.merges arr[str,446189]  = ["Ġ Ġ", "Ġ ĠĠĠ", "ĠĠ ĠĠ", "...
llama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 199999
llama_model_loader: - kv  29:                    tokenizer.chat_template str              = {#-\n  In addition to the normal input...
llama_model_loader: - kv  30:               general.quantization_version u32              = 2
llama_model_loader: - kv  31:                          general.file_type u32              = 38
llama_model_loader: - kv  32:                                   split.no u16              = 0
llama_model_loader: - kv  33:                        split.tensors.count i32              = 687
llama_model_loader: - kv  34:                                split.count u16              = 3
llama_model_loader: - type  f32:  433 tensors
llama_model_loader: - type q8_0:  146 tensors
llama_model_loader: - type mxfp4:  108 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = MXFP4 MoE
print_info: file size   = 59.02 GiB (4.34 BPW) 
load: printing all EOG tokens:
load:   - 199999 ('<|endoftext|>')
load:   - 200002 ('<|return|>')
load:   - 200007 ('<|end|>')
load:   - 200012 ('<|call|>')
load: special_eog_ids contains both '<|return|>' and '<|call|>' tokens, removing '<|end|>' token from EOG list
load: special tokens cache size = 21
load: token to piece cache size = 1.3332 MB
print_info: arch             = gpt-oss
print_info: vocab_only       = 0
print_info: n_ctx_train      = 131072
print_info: n_embd           = 2880
print_info: n_layer          = 36
print_info: n_head           = 64
print_info: n_head_kv        = 8
print_info: n_rot            = 64
print_info: n_swa            = 128
print_info: is_swa_any       = 1
print_info: n_embd_head_k    = 64
print_info: n_embd_head_v    = 64
print_info: n_gqa            = 8
print_info: n_embd_k_gqa     = 512
print_info: n_embd_v_gqa     = 512
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-05
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 2880
print_info: n_expert         = 128
print_info: n_expert_used    = 4
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 2
print_info: rope scaling     = yarn
print_info: freq_base_train  = 150000.0
print_info: freq_scale_train = 0.03125
print_info: n_ctx_orig_yarn  = 4096
print_info: rope_finetuned   = unknown
print_info: model type       = ?B
print_info: model params     = 116.83 B
print_info: general.name     = Prerelease 100 120b Hf
print_info: n_ff_exp         = 2880
print_info: vocab type       = BPE
print_info: n_vocab          = 201088
print_info: n_merges         = 446189
print_info: BOS token        = 11 ','
print_info: EOS token        = 199999 '<|endoftext|>'
print_info: EOT token        = 199999 '<|endoftext|>'
print_info: LF token         = 198 'Ċ'
print_info: EOG token        = 199999 '<|endoftext|>'
print_info: EOG token        = 200002 '<|return|>'
print_info: EOG token        = 200012 '<|call|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 36 repeating layers to GPU
load_tensors: offloaded 36/37 layers to GPU
load_tensors:        CUDA0 model buffer size = 14816.21 MiB
load_tensors:        CUDA1 model buffer size = 14816.21 MiB
load_tensors:        CUDA2 model buffer size = 14816.21 MiB
load_tensors:        CUDA3 model buffer size = 14816.21 MiB
load_tensors:   CPU_Mapped model buffer size =  1173.66 MiB
....................................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 131072
llama_context: n_ctx_per_seq = 131072
llama_context: n_batch       = 256
llama_context: n_ubatch      = 256
llama_context: causal_attn   = 1
llama_context: flash_attn    = 0
llama_context: kv_unified    = false
llama_context: freq_base     = 150000.0
llama_context: freq_scale    = 0.03125
llama_context:        CPU  output buffer size =     0.77 MiB
llama_kv_cache_unified_iswa: creating non-SWA KV cache, size = 131072 cells
llama_kv_cache_unified:      CUDA0 KV buffer size =  1024.00 MiB
llama_kv_cache_unified:      CUDA1 KV buffer size =  1280.00 MiB
llama_kv_cache_unified:      CUDA2 KV buffer size =  1024.00 MiB
llama_kv_cache_unified:      CUDA3 KV buffer size =  1280.00 MiB
llama_kv_cache_unified: size = 4608.00 MiB (131072 cells,  18 layers,  1/1 seqs), K (f16): 2304.00 MiB, V (f16): 2304.00 MiB
llama_kv_cache_unified_iswa: creating     SWA KV cache, size = 384 cells
llama_kv_cache_unified:      CUDA0 KV buffer size =     3.75 MiB
llama_kv_cache_unified:      CUDA1 KV buffer size =     3.00 MiB
llama_kv_cache_unified:      CUDA2 KV buffer size =     3.75 MiB
llama_kv_cache_unified:      CUDA3 KV buffer size =     3.00 MiB
llama_kv_cache_unified: size =   13.50 MiB (   384 cells,  18 layers,  1/1 seqs), K (f16):    6.75 MiB, V (f16):    6.75 MiB
llama_context:      CUDA0 compute buffer size =  8336.00 MiB
llama_context:      CUDA1 compute buffer size =  8336.00 MiB
llama_context:      CUDA2 compute buffer size =  8336.00 MiB
llama_context:      CUDA3 compute buffer size =  8336.01 MiB
llama_context:  CUDA_Host compute buffer size =   133.19 MiB
llama_context: graph nodes  = 2166
llama_context: graph splits = 7 (with bs=256), 6 (with bs=1)
common_init_from_params: KV cache shifting is not supported for this context, disabling KV cache shifting
common_init_from_params: added <|endoftext|> logit bias = -inf
common_init_from_params: added <|return|> logit bias = -inf
common_init_from_params: added <|call|> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 131072
srv          init: initializing slots, n_slots = 1
slot         init: id  0 | task -1 | new slot n_ctx_slot = 131072
main: model loaded
main: chat template, chat_template: {#-

Operating systems

Linux

GGML backends

CUDA

Hardware

Threadripper 7965WX, 4x RTX 5090

Models

No response

Problem description & steps to reproduce

Launch model, navigate ot llama-server webui, paste 40k context prompt

First Bad Commit

fd1234c

Relevant log output

srv  update_slots: all slots are idle
srv  params_from_: Chat format: Content-only
slot launch_slot_: id  0 | task 0 | processing task
slot update_slots: id  0 | task 0 | new prompt, n_ctx_slot = 131072, n_keep = 0, n_prompt_tokens = 41757
slot update_slots: id  0 | task 0 | kv cache rm [0, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 256, n_tokens = 256, progress = 0.006131
slot update_slots: id  0 | task 0 | kv cache rm [256, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 512, n_tokens = 256, progress = 0.012261
slot update_slots: id  0 | task 0 | kv cache rm [512, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 768, n_tokens = 256, progress = 0.018392
slot update_slots: id  0 | task 0 | kv cache rm [768, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 1024, n_tokens = 256, progress = 0.024523
slot update_slots: id  0 | task 0 | kv cache rm [1024, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 1280, n_tokens = 256, progress = 0.030654
slot update_slots: id  0 | task 0 | kv cache rm [1280, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 1536, n_tokens = 256, progress = 0.036784
slot update_slots: id  0 | task 0 | kv cache rm [1536, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 1792, n_tokens = 256, progress = 0.042915
slot update_slots: id  0 | task 0 | kv cache rm [1792, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 2048, n_tokens = 256, progress = 0.049046
slot update_slots: id  0 | task 0 | kv cache rm [2048, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 2304, n_tokens = 256, progress = 0.055176
slot update_slots: id  0 | task 0 | kv cache rm [2304, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 2560, n_tokens = 256, progress = 0.061307
slot update_slots: id  0 | task 0 | kv cache rm [2560, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 2816, n_tokens = 256, progress = 0.067438
slot update_slots: id  0 | task 0 | kv cache rm [2816, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 3072, n_tokens = 256, progress = 0.073569
slot update_slots: id  0 | task 0 | kv cache rm [3072, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 3328, n_tokens = 256, progress = 0.079699
slot update_slots: id  0 | task 0 | kv cache rm [3328, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 3584, n_tokens = 256, progress = 0.085830
slot update_slots: id  0 | task 0 | kv cache rm [3584, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 3840, n_tokens = 256, progress = 0.091961
slot update_slots: id  0 | task 0 | kv cache rm [3840, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 4096, n_tokens = 256, progress = 0.098091
slot update_slots: id  0 | task 0 | kv cache rm [4096, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 4352, n_tokens = 256, progress = 0.104222
slot update_slots: id  0 | task 0 | kv cache rm [4352, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 4608, n_tokens = 256, progress = 0.110353
slot update_slots: id  0 | task 0 | kv cache rm [4608, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 4864, n_tokens = 256, progress = 0.116483
slot update_slots: id  0 | task 0 | kv cache rm [4864, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 5120, n_tokens = 256, progress = 0.122614
slot update_slots: id  0 | task 0 | kv cache rm [5120, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 5376, n_tokens = 256, progress = 0.128745
slot update_slots: id  0 | task 0 | kv cache rm [5376, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 5632, n_tokens = 256, progress = 0.134876
slot update_slots: id  0 | task 0 | kv cache rm [5632, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 5888, n_tokens = 256, progress = 0.141006
slot update_slots: id  0 | task 0 | kv cache rm [5888, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 6144, n_tokens = 256, progress = 0.147137
slot update_slots: id  0 | task 0 | kv cache rm [6144, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 6400, n_tokens = 256, progress = 0.153268
slot update_slots: id  0 | task 0 | kv cache rm [6400, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 6656, n_tokens = 256, progress = 0.159398
slot update_slots: id  0 | task 0 | kv cache rm [6656, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 6912, n_tokens = 256, progress = 0.165529
slot update_slots: id  0 | task 0 | kv cache rm [6912, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 7168, n_tokens = 256, progress = 0.171660
slot update_slots: id  0 | task 0 | kv cache rm [7168, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 7424, n_tokens = 256, progress = 0.177791
slot update_slots: id  0 | task 0 | kv cache rm [7424, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 7680, n_tokens = 256, progress = 0.183921
slot update_slots: id  0 | task 0 | kv cache rm [7680, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 7936, n_tokens = 256, progress = 0.190052
slot update_slots: id  0 | task 0 | kv cache rm [7936, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 8192, n_tokens = 256, progress = 0.196183
slot update_slots: id  0 | task 0 | kv cache rm [8192, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 8448, n_tokens = 256, progress = 0.202313
slot update_slots: id  0 | task 0 | kv cache rm [8448, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 8704, n_tokens = 256, progress = 0.208444
slot update_slots: id  0 | task 0 | kv cache rm [8704, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 8960, n_tokens = 256, progress = 0.214575
slot update_slots: id  0 | task 0 | kv cache rm [8960, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 9216, n_tokens = 256, progress = 0.220706
slot update_slots: id  0 | task 0 | kv cache rm [9216, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 9472, n_tokens = 256, progress = 0.226836
slot update_slots: id  0 | task 0 | kv cache rm [9472, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 9728, n_tokens = 256, progress = 0.232967
slot update_slots: id  0 | task 0 | kv cache rm [9728, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 9984, n_tokens = 256, progress = 0.239098
slot update_slots: id  0 | task 0 | kv cache rm [9984, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 10240, n_tokens = 256, progress = 0.245228
slot update_slots: id  0 | task 0 | kv cache rm [10240, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 10496, n_tokens = 256, progress = 0.251359
slot update_slots: id  0 | task 0 | kv cache rm [10496, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 10752, n_tokens = 256, progress = 0.257490
slot update_slots: id  0 | task 0 | kv cache rm [10752, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 11008, n_tokens = 256, progress = 0.263620
slot update_slots: id  0 | task 0 | kv cache rm [11008, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 11264, n_tokens = 256, progress = 0.269751
slot update_slots: id  0 | task 0 | kv cache rm [11264, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 11520, n_tokens = 256, progress = 0.275882
slot update_slots: id  0 | task 0 | kv cache rm [11520, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 11776, n_tokens = 256, progress = 0.282013
slot update_slots: id  0 | task 0 | kv cache rm [11776, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 12032, n_tokens = 256, progress = 0.288143
slot update_slots: id  0 | task 0 | kv cache rm [12032, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 12288, n_tokens = 256, progress = 0.294274
slot update_slots: id  0 | task 0 | kv cache rm [12288, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 12544, n_tokens = 256, progress = 0.300405
slot update_slots: id  0 | task 0 | kv cache rm [12544, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 12800, n_tokens = 256, progress = 0.306535
slot update_slots: id  0 | task 0 | kv cache rm [12800, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 13056, n_tokens = 256, progress = 0.312666
slot update_slots: id  0 | task 0 | kv cache rm [13056, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 13312, n_tokens = 256, progress = 0.318797
slot update_slots: id  0 | task 0 | kv cache rm [13312, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 13568, n_tokens = 256, progress = 0.324928
slot update_slots: id  0 | task 0 | kv cache rm [13568, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 13824, n_tokens = 256, progress = 0.331058
slot update_slots: id  0 | task 0 | kv cache rm [13824, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 14080, n_tokens = 256, progress = 0.337189
slot update_slots: id  0 | task 0 | kv cache rm [14080, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 14336, n_tokens = 256, progress = 0.343320
slot update_slots: id  0 | task 0 | kv cache rm [14336, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 14592, n_tokens = 256, progress = 0.349450
slot update_slots: id  0 | task 0 | kv cache rm [14592, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 14848, n_tokens = 256, progress = 0.355581
slot update_slots: id  0 | task 0 | kv cache rm [14848, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 15104, n_tokens = 256, progress = 0.361712
slot update_slots: id  0 | task 0 | kv cache rm [15104, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 15360, n_tokens = 256, progress = 0.367843
slot update_slots: id  0 | task 0 | kv cache rm [15360, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 15616, n_tokens = 256, progress = 0.373973
slot update_slots: id  0 | task 0 | kv cache rm [15616, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 15872, n_tokens = 256, progress = 0.380104
slot update_slots: id  0 | task 0 | kv cache rm [15872, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 16128, n_tokens = 256, progress = 0.386235
slot update_slots: id  0 | task 0 | kv cache rm [16128, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 16384, n_tokens = 256, progress = 0.392365
slot update_slots: id  0 | task 0 | kv cache rm [16384, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 16640, n_tokens = 256, progress = 0.398496
slot update_slots: id  0 | task 0 | kv cache rm [16640, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 16896, n_tokens = 256, progress = 0.404627
slot update_slots: id  0 | task 0 | kv cache rm [16896, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 17152, n_tokens = 256, progress = 0.410757
slot update_slots: id  0 | task 0 | kv cache rm [17152, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 17408, n_tokens = 256, progress = 0.416888
slot update_slots: id  0 | task 0 | kv cache rm [17408, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 17664, n_tokens = 256, progress = 0.423019
slot update_slots: id  0 | task 0 | kv cache rm [17664, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 17920, n_tokens = 256, progress = 0.429150
slot update_slots: id  0 | task 0 | kv cache rm [17920, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 18176, n_tokens = 256, progress = 0.435280
slot update_slots: id  0 | task 0 | kv cache rm [18176, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 18432, n_tokens = 256, progress = 0.441411
slot update_slots: id  0 | task 0 | kv cache rm [18432, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 18688, n_tokens = 256, progress = 0.447542
slot update_slots: id  0 | task 0 | kv cache rm [18688, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 18944, n_tokens = 256, progress = 0.453672
slot update_slots: id  0 | task 0 | kv cache rm [18944, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 19200, n_tokens = 256, progress = 0.459803
slot update_slots: id  0 | task 0 | kv cache rm [19200, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 19456, n_tokens = 256, progress = 0.465934
slot update_slots: id  0 | task 0 | kv cache rm [19456, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 19712, n_tokens = 256, progress = 0.472065
slot update_slots: id  0 | task 0 | kv cache rm [19712, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 19968, n_tokens = 256, progress = 0.478195
slot update_slots: id  0 | task 0 | kv cache rm [19968, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 20224, n_tokens = 256, progress = 0.484326
slot update_slots: id  0 | task 0 | kv cache rm [20224, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 20480, n_tokens = 256, progress = 0.490457
slot update_slots: id  0 | task 0 | kv cache rm [20480, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 20736, n_tokens = 256, progress = 0.496587
slot update_slots: id  0 | task 0 | kv cache rm [20736, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 20992, n_tokens = 256, progress = 0.502718
slot update_slots: id  0 | task 0 | kv cache rm [20992, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 21248, n_tokens = 256, progress = 0.508849
slot update_slots: id  0 | task 0 | kv cache rm [21248, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 21504, n_tokens = 256, progress = 0.514980
slot update_slots: id  0 | task 0 | kv cache rm [21504, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 21760, n_tokens = 256, progress = 0.521110
slot update_slots: id  0 | task 0 | kv cache rm [21760, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 22016, n_tokens = 256, progress = 0.527241
slot update_slots: id  0 | task 0 | kv cache rm [22016, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 22272, n_tokens = 256, progress = 0.533372
slot update_slots: id  0 | task 0 | kv cache rm [22272, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 22528, n_tokens = 256, progress = 0.539502
slot update_slots: id  0 | task 0 | kv cache rm [22528, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 22784, n_tokens = 256, progress = 0.545633
slot update_slots: id  0 | task 0 | kv cache rm [22784, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 23040, n_tokens = 256, progress = 0.551764
slot update_slots: id  0 | task 0 | kv cache rm [23040, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 23296, n_tokens = 256, progress = 0.557894
slot update_slots: id  0 | task 0 | kv cache rm [23296, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 23552, n_tokens = 256, progress = 0.564025
slot update_slots: id  0 | task 0 | kv cache rm [23552, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 23808, n_tokens = 256, progress = 0.570156
slot update_slots: id  0 | task 0 | kv cache rm [23808, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 24064, n_tokens = 256, progress = 0.576287
slot update_slots: id  0 | task 0 | kv cache rm [24064, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 24320, n_tokens = 256, progress = 0.582417
slot update_slots: id  0 | task 0 | kv cache rm [24320, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 24576, n_tokens = 256, progress = 0.588548
slot update_slots: id  0 | task 0 | kv cache rm [24576, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 24832, n_tokens = 256, progress = 0.594679
slot update_slots: id  0 | task 0 | kv cache rm [24832, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 25088, n_tokens = 256, progress = 0.600809
slot update_slots: id  0 | task 0 | kv cache rm [25088, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 25344, n_tokens = 256, progress = 0.606940
slot update_slots: id  0 | task 0 | kv cache rm [25344, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 25600, n_tokens = 256, progress = 0.613071
slot update_slots: id  0 | task 0 | kv cache rm [25600, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 25856, n_tokens = 256, progress = 0.619202
slot update_slots: id  0 | task 0 | kv cache rm [25856, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 26112, n_tokens = 256, progress = 0.625332
slot update_slots: id  0 | task 0 | kv cache rm [26112, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 26368, n_tokens = 256, progress = 0.631463
slot update_slots: id  0 | task 0 | kv cache rm [26368, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 26624, n_tokens = 256, progress = 0.637594
slot update_slots: id  0 | task 0 | kv cache rm [26624, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 26880, n_tokens = 256, progress = 0.643724
slot update_slots: id  0 | task 0 | kv cache rm [26880, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 27136, n_tokens = 256, progress = 0.649855
slot update_slots: id  0 | task 0 | kv cache rm [27136, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 27392, n_tokens = 256, progress = 0.655986
slot update_slots: id  0 | task 0 | kv cache rm [27392, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 27648, n_tokens = 256, progress = 0.662117
slot update_slots: id  0 | task 0 | kv cache rm [27648, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 27904, n_tokens = 256, progress = 0.668247
slot update_slots: id  0 | task 0 | kv cache rm [27904, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 28160, n_tokens = 256, progress = 0.674378
slot update_slots: id  0 | task 0 | kv cache rm [28160, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 28416, n_tokens = 256, progress = 0.680509
slot update_slots: id  0 | task 0 | kv cache rm [28416, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 28672, n_tokens = 256, progress = 0.686639
slot update_slots: id  0 | task 0 | kv cache rm [28672, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 28928, n_tokens = 256, progress = 0.692770
slot update_slots: id  0 | task 0 | kv cache rm [28928, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 29184, n_tokens = 256, progress = 0.698901
slot update_slots: id  0 | task 0 | kv cache rm [29184, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 29440, n_tokens = 256, progress = 0.705032
slot update_slots: id  0 | task 0 | kv cache rm [29440, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 29696, n_tokens = 256, progress = 0.711162
slot update_slots: id  0 | task 0 | kv cache rm [29696, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 29952, n_tokens = 256, progress = 0.717293
slot update_slots: id  0 | task 0 | kv cache rm [29952, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 30208, n_tokens = 256, progress = 0.723424
slot update_slots: id  0 | task 0 | kv cache rm [30208, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 30464, n_tokens = 256, progress = 0.729554
slot update_slots: id  0 | task 0 | kv cache rm [30464, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 30720, n_tokens = 256, progress = 0.735685
slot update_slots: id  0 | task 0 | kv cache rm [30720, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 30976, n_tokens = 256, progress = 0.741816
slot update_slots: id  0 | task 0 | kv cache rm [30976, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 31232, n_tokens = 256, progress = 0.747946
slot update_slots: id  0 | task 0 | kv cache rm [31232, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 31488, n_tokens = 256, progress = 0.754077
slot update_slots: id  0 | task 0 | kv cache rm [31488, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 31744, n_tokens = 256, progress = 0.760208
slot update_slots: id  0 | task 0 | kv cache rm [31744, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 32000, n_tokens = 256, progress = 0.766339
slot update_slots: id  0 | task 0 | kv cache rm [32000, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 32256, n_tokens = 256, progress = 0.772469
slot update_slots: id  0 | task 0 | kv cache rm [32256, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 32512, n_tokens = 256, progress = 0.778600
slot update_slots: id  0 | task 0 | kv cache rm [32512, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 32768, n_tokens = 256, progress = 0.784731
slot update_slots: id  0 | task 0 | kv cache rm [32768, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 33024, n_tokens = 256, progress = 0.790861
slot update_slots: id  0 | task 0 | kv cache rm [33024, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 33280, n_tokens = 256, progress = 0.796992
slot update_slots: id  0 | task 0 | kv cache rm [33280, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 33536, n_tokens = 256, progress = 0.803123
slot update_slots: id  0 | task 0 | kv cache rm [33536, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 33792, n_tokens = 256, progress = 0.809254
slot update_slots: id  0 | task 0 | kv cache rm [33792, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 34048, n_tokens = 256, progress = 0.815384
slot update_slots: id  0 | task 0 | kv cache rm [34048, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 34304, n_tokens = 256, progress = 0.821515
slot update_slots: id  0 | task 0 | kv cache rm [34304, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 34560, n_tokens = 256, progress = 0.827646
slot update_slots: id  0 | task 0 | kv cache rm [34560, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 34816, n_tokens = 256, progress = 0.833776
slot update_slots: id  0 | task 0 | kv cache rm [34816, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 35072, n_tokens = 256, progress = 0.839907
slot update_slots: id  0 | task 0 | kv cache rm [35072, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 35328, n_tokens = 256, progress = 0.846038
slot update_slots: id  0 | task 0 | kv cache rm [35328, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 35584, n_tokens = 256, progress = 0.852169
slot update_slots: id  0 | task 0 | kv cache rm [35584, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 35840, n_tokens = 256, progress = 0.858299
slot update_slots: id  0 | task 0 | kv cache rm [35840, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 36096, n_tokens = 256, progress = 0.864430
slot update_slots: id  0 | task 0 | kv cache rm [36096, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 36352, n_tokens = 256, progress = 0.870561
slot update_slots: id  0 | task 0 | kv cache rm [36352, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 36608, n_tokens = 256, progress = 0.876691
slot update_slots: id  0 | task 0 | kv cache rm [36608, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 36864, n_tokens = 256, progress = 0.882822
slot update_slots: id  0 | task 0 | kv cache rm [36864, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 37120, n_tokens = 256, progress = 0.888953
slot update_slots: id  0 | task 0 | kv cache rm [37120, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 37376, n_tokens = 256, progress = 0.895083
slot update_slots: id  0 | task 0 | kv cache rm [37376, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 37632, n_tokens = 256, progress = 0.901214
slot update_slots: id  0 | task 0 | kv cache rm [37632, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 37888, n_tokens = 256, progress = 0.907345
slot update_slots: id  0 | task 0 | kv cache rm [37888, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 38144, n_tokens = 256, progress = 0.913476
slot update_slots: id  0 | task 0 | kv cache rm [38144, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 38400, n_tokens = 256, progress = 0.919606
slot update_slots: id  0 | task 0 | kv cache rm [38400, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 38656, n_tokens = 256, progress = 0.925737
slot update_slots: id  0 | task 0 | kv cache rm [38656, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 38912, n_tokens = 256, progress = 0.931868
slot update_slots: id  0 | task 0 | kv cache rm [38912, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 39168, n_tokens = 256, progress = 0.937998
slot update_slots: id  0 | task 0 | kv cache rm [39168, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 39424, n_tokens = 256, progress = 0.944129
slot update_slots: id  0 | task 0 | kv cache rm [39424, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 39680, n_tokens = 256, progress = 0.950260
slot update_slots: id  0 | task 0 | kv cache rm [39680, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 39936, n_tokens = 256, progress = 0.956391
slot update_slots: id  0 | task 0 | kv cache rm [39936, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 40192, n_tokens = 256, progress = 0.962521
slot update_slots: id  0 | task 0 | kv cache rm [40192, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 40448, n_tokens = 256, progress = 0.968652
slot update_slots: id  0 | task 0 | kv cache rm [40448, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 40704, n_tokens = 256, progress = 0.974783
slot update_slots: id  0 | task 0 | kv cache rm [40704, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 40960, n_tokens = 256, progress = 0.980913
slot update_slots: id  0 | task 0 | kv cache rm [40960, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 41216, n_tokens = 256, progress = 0.987044
slot update_slots: id  0 | task 0 | kv cache rm [41216, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 41472, n_tokens = 256, progress = 0.993175
slot update_slots: id  0 | task 0 | kv cache rm [41472, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 41728, n_tokens = 256, progress = 0.999305
slot update_slots: id  0 | task 0 | kv cache rm [41728, end)
slot update_slots: id  0 | task 0 | prompt processing progress, n_past = 41757, n_tokens = 29, progress = 1.000000
slot update_slots: id  0 | task 0 | prompt done, n_past = 41757, n_tokens = 29
srv  cancel_tasks: cancel task, id_task = 0
srv  log_server_r: request: POST /v1/chat/completions 127.0.0.1 200
slot      release: id  0 | task 0 | stop processing: n_past = 45853, truncated = 0
srv  update_slots: all slots are idle

[CISC]

As established in GLM-4.5 PR this is an issue when batch and ubatch are both 4096 (possibly other sizes as well), not specific to GPT-OSS.

[thad0ctor]

As established in GLM-4.5 PR this is an issue when batch and ubatch are both 4096 (possibly other sizes as well), not specific to GPT-OSS.

Thank you, interesting. I tested to see if this was the underlying cause. I kept mostly the same launch parameters just swapping models (and offloaded layers for GLM), only removing the batch and ubatch parameters

base parameters:

llama-server -m XXXX.gguf --threads 24 --threads-batch 48 --ctx-size 131072 --temp 0.6 --min-p 0.01 --tensor-split 23,23,24,24 --n-gpu-layers XXX --main-gpu 1 --flash-attn --port 5001 --no-warmup

Scenario 1: GPT OSS - native - removed batch and ubatch arguments - still get gibberish with 40k prompt
Scenario 2: Qwen3 Coder 32B Q_8 - removed batch and ubatch arguments - no gibberish, works as intended with 40k prompt
Scenario 3: GLM Air Q6- removed batch and ubatch arguments - no gibberish in response either

[David-AU-github]

GGUFs except IQ4_NL, Q5_1, "moe gguf" (new) and Q8 work ; others (ggufs/imat ggufs) are borked due to tensor size/fallback issues.

[thad0ctor]

GGUFs except IQ4_NL, Q5_1, "moe gguf" (new) and Q8 work ; others (ggufs/imat ggufs) are borked due to tensor size/fallback issues.

I have only tested with the native and F16 gguf, this error happens with GGML's gguf that they specifically linked in the PR adding support for GPT OSS

[ajunca]

I can confirm is happening too with GLM 4.5 Air with gguf (Q2, Q3, Q4) and mxfp4. On different batch options, I can not pin point exactly which ones, but for example 512 batch and 512 ubatch.

EDIT: Also with defaults (2048 batch 512 ubatch). Not sure what values to use...

[CISC]

@ajunca If you get nonsense with default options something is either wrong with your setup or the GGUF, but I'm also curious why you mention MXFP4 with GLM-4.5 Air?

[ajunca]

@CISC I'm not getting nonsense always, is this "GGGGGGGGGGGGGGGGGGG..." Every now and then. Tried many different GLM-4.5 Air (unsloth, lovedheart, bartowski, etc) with different quants. All does this at some points (not always), its a bit random. But some configurations more often than others. I'm suspecting the actual context size used and the size of batch-ubatch.

If I leave the "GGG..." run long enough it ends with a CUDA memory crash for access outside a buffer:

/app/ggml/src/ggml-cuda/ggml-cuda.cu:84: CUDA error
CUDA error: an illegal memory access was encountered
[0m  current device: 1, in function ggml_backend_cuda_synchronize at /app/ggml/src/ggml-cuda/ggml-cuda.cu:2612
[0m  cudaStreamSynchronize(cuda_ctx->stream())
[0mlibggml-base.so(+0x16d4b)[0x7fb29ab44d4b]
libggml-base.so(ggml_print_backtrace+0x21f)[0x7fb29ab451af]
libggml-base.so(ggml_abort+0x152)[0x7fb29ab45382]
/app/libggml-cuda.so(+0xea826)[0x7fb292aea826]
/app/libggml-cuda.so(+0xebaf3)[0x7fb292aebaf3]
libggml-base.so(ggml_backend_sched_synchronize+0x2e)[0x7fb29ab5aece]
libllama.so(_ZN13llama_context11synchronizeEv+0x14)[0x7fb29a897ac4]
libllama.so(llama_get_logits_ith+0x16)[0x7fb29a899de6]
/app/llama-server(+0x1f00fe)[0x55d97e53e0fe]
/app/llama-server(+0xdf547)[0x55d97e42d547]
/app/llama-server(+0x857bd)[0x55d97e3d37bd]
/app/llama-server(+0x4d6a5)[0x55d97e39b6a5]
/usr/lib/x86_64-linux-gnu/libc.so.6(+0x29d90)[0x7fb29a029d90]
/usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80)[0x7fb29a029e40]
/app/llama-server(+0x4f0f5)[0x55d97e39d0f5]

I mention MXFP4 (lovedheart version) because is by quite a lot, the one giving best results in my case for the amount of memory/compute is using.

Here one example of such running models:

--model /app/models/lovedheart/GLM-4.5-Air-MXFP4_MOE-00001-of-00005.gguf
-ctk q8_0 -ctv q5_1
--flash-attn
--tensor-split 22,26
--main-gpu 1
-ot "\.(0|1|2|3|4|5|6|7|8)\.ffn_(down|up|gate)_exps.=CPU"
-ot "\.(24|25|26|27|28|29|30|31|32)\.ffn_(down|up|gate)_exps.=CPU"
--n-gpu-layers 48
--ctx-size 61440
--temp 0.6
--top-k 40
--top-p 0.95
--min-p 0.0
--batch-size 512
--ubatch-size 512
--jinja
--log-colors
--metrics

EDIT: Tried also different KV quants, and also happens this.

[CISC]

If I leave the "GGG..." run long enough it ends with a CUDA memory crash for access outside a buffer:

Ok, that's interesting.

I mention MXFP4 (lovedheart version) because is by quite a lot, the one giving best results in my case for the amount of memory/compute is using.

I would have thought it generally does a bad job at quantizing this model, though GLM-4.5 (Air) does seem to be able to withstand fairly brutal quants.

[CISC]

EDIT: Tried also different KV quants, and also happens this.

Did you try without?

[CISC]

Also try without Flash Attention.

[ajunca]

@CISC I tried your recommendations (and some more):

• KV quantization (or no quant): same problem
• Disable Flash Attention: same problem
• Tried with official docker container (using CUDA version 12.4.0), and also tried with custom build (using CUDA version 12.9.0 and GGML_CUDA_FA_ALL_QUANTS=ON): same problem
• All moe layers to CPU: same problem
• Use Vulkan backend (instate of CUDA): it works (but like 10% of speed of CUDA).

So maybe is some kind of CUDA problem? @thad0ctor have you tried with Vulkan backend?

EDIT: Checking dmesg | grep -i pciis confirming the memory issue, tried to read illegal space:

[13838.025927] NVRM: Xid (PCI:0000:0a:00): 31, pid=146294, name=llama-server, Ch 00000018, intr 00000000. MMU Fault: ENGINE GRAPHICS GPC3 GPCCLIENT_T1_7 faulted @ 0x7e4b_ebef2000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ

EDIT2: Checked also for PCI-E errors nvidia-smi dmon -s et -d 10 -o DT (in case of a hardware problem), but there are non.

EDIT3: I think the chances of happening increases as the input context gets bigger and the batch/ubatch smaller.

[ajunca]

If its of any help, I'm trying to debug a bit deeper with compute-sanitizer --tool=memcheck and got:

llama-swap  | ========= Program hit cudaErrorGraphExecUpdateFailure (error 910) due to "the graph update was not performed because it included changes which violated constraints specific to instantiated graph update" on CUDA API call to cudaGraphExecUpdate.
llama-swap  | =========     Saved host backtrace up to driver entry point at error
llama-swap  | =========         Host Frame: ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) [0xf92ea] in libggml-cuda.so
llama-swap  | =========         Host Frame: ggml_backend_sched_graph_compute_async [0x2fd5c] in libggml-base.so
llama-swap  | =========         Host Frame: llama_context::graph_compute(ggml_cgraph*, bool) [0x987d8] in libllama.so
llama-swap  | =========         Host Frame: llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) [0x98b84] in libllama.so
llama-swap  | =========         Host Frame: llama_context::decode(llama_batch const&) [0x9f1f3] in libllama.so
llama-swap  | =========         Host Frame: llama_decode [0xa013f] in libllama.so
llama-swap  | =========         Host Frame: server_context::update_slots() [0xddf00] in llama-server
llama-swap  | =========         Host Frame: server_queue::start_loop() [0x857bc] in llama-server
llama-swap  | =========         Host Frame: main [0x4d6a4] in llama-server
llama-swap  | ========= 
llama-swap  | ========= Program hit cudaErrorGraphExecUpdateFailure (error 910) due to "the graph update was not performed because it included changes which violated constraints specific to instantiated graph update" on CUDA API call to cudaGetLastError.
llama-swap  | =========     Saved host backtrace up to driver entry point at error
llama-swap  | =========         Host Frame: ggml_backend_cuda_graph_compute(ggml_backend*, ggml_cgraph*) [0xfad06] in libggml-cuda.so
llama-swap  | =========         Host Frame: ggml_backend_sched_graph_compute_async [0x2fd5c] in libggml-base.so
llama-swap  | =========         Host Frame: llama_context::graph_compute(ggml_cgraph*, bool) [0x987d8] in libllama.so
llama-swap  | =========         Host Frame: llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&) [0x98b84] in libllama.so
llama-swap  | =========         Host Frame: llama_context::decode(llama_batch const&) [0x9f1f3] in libllama.so
llama-swap  | =========         Host Frame: llama_decode [0xa013f] in libllama.so
llama-swap  | =========         Host Frame: server_context::update_slots() [0xddf00] in llama-server
llama-swap  | =========         Host Frame: server_queue::start_loop() [0x857bc] in llama-server
llama-swap  | =========         Host Frame: main [0x4d6a4] in llama-server
llama-swap  | ========= 

[ajunca]

After seeing the cudaErrorGraphExecUpdateFailure error that I posted last, I compiled llama.cpp with GGML_CUDA_USE_GRAPHS=OFF and the issue seems to be solved. I should do some more extensive tests, but it is already late at night, and the one was always failing is now not (no more GGGGGGGGGGGGGGGGGGG...).

I think this is a bug, maybe we should open a proper bug report? Or at least maybe change the title on this one, as it is affecting more models than GPT-OSS. Or what you reckon?

As the error points Program hit cudaErrorGraphExecUpdateFailure (error 910) due to "the graph update was not performed because it included changes which violated constraints specific to instantiated graph update" on CUDA API call to cudaGetLastError., I think at some point the program is trying to update the CUDA graph with invalid arguments. Maybe a bigger ubatch size than was initially constructed? llama_context::process_ubatch(llama_ubatch const&, llm_graph_type, llama_memory_context_i*, ggml_status&)