Misc. bug: Reverse port not properly closed after SSE stream is closed

[jpm-canonical]

Name and Version

$ ./llama-server --version
load_backend: loaded CPU backend from /home/jpmeijers/llama.cpp/build/bin/libggml-cpu-haswell.so
version: 8091 (238856e)
built with GNU 15.2.0 for Linux x86_64

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

llama-server --model gemma-3-4b-it-q4_0.gguf --port 8080 --host 0.0.0.0

Problem description & steps to reproduce

When prompting a model using the OpenAI compatible API, using streaming mode, the reverse socket does not seem to be closed. If llama-server is closed, the socket still remains, blocking the use of the same port number until the TIME_WAIT timeout of the OS has passed.

To reproduce:

1. Start llama-server, for example using this command:

   ~/llama.cpp/build/bin$ ./llama-server --model gemma-3-4b-it-q4_0.gguf --port 8080 --host 0.0.0.0
   

2. Use curl to prompt the model, using streaming mode:

   curl http://localhost:8080/v1/chat/completions \
     -N -H "Accept: text/event-stream" \
     -H "Content-Type: application/json" \
     -d '{
       "model": "gemma-3-4b-it-ov-int4-fq",
       "messages": [
         {
           "role": "system",
           "content": "You are a helpful assistant."
         },
         {
           "role": "user",
           "content": "Write a short poem about coding."
         }
       ],
       "stream": true
     }'
   

3. Wait until curl has received all the chunks and has exited.

4. Quit llama-server.

5. Use socat to try and bind to the same port as llama-server used. It will fail with an error:

   $ socat TCP4-LISTEN:8080,bind=0.0.0.0,reuseaddr,fork STDIO
   2026/02/20 14:24:01 socat[96723] E bind(5, {AF=2 0.0.0.0:8080}, 16): Address already in use
   

   Or try running llama-server again:

   $ ./llama-server --model gemma-3-4b-it-q4_0.gguf --port 8080 --host 0.0.0.0
   main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
   build: 8120 (0e93d6e45) with GNU 15.2.0 for Linux x86_64
   system info: n_threads = 4, n_threads_batch = 4, total_threads = 8
   
   system_info: n_threads = 4 (n_threads_batch = 4) / 8 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 
   
   Running without SSL
   init: using 7 threads for HTTP server
   start: binding port with default address family
   start: couldn't bind HTTP server socket, hostname: 0.0.0.0, port: 8080
   srv    operator(): operator(): cleaning up before exit...
   main: exiting due to HTTP server error
   

6. Use netstat to see there is a dangling port 8080 in TIME_WAIT state:

   $ sudo netstat -anp | grep ':8080'
   tcp        0      0 127.0.0.1:8080          127.0.0.1:49154         TIME_WAIT   -                   
   

The same steps can be repeated, without using streaming mode. In that case the sockets are cleaned up correctly when llama-server exits, netcat will show no sockets in use, and socat can immediately bind to the same port.

First Bad Commit

I have seen this issue is earlier 7000's builds too, but not sure when it started

Relevant log output

Logs

$ ./llama-server --model /snap/gemma3/components/mnt/model-4b-it-q4-0-gguf/157/gemma-3-4b-it-q4_0.gguf --port 8080 --host 0.0.0.0
load_backend: loaded CPU backend from /home/jpmeijers/llama.cpp/build/bin/libggml-cpu-haswell.so
main: n_parallel is set to auto, using n_parallel = 4 and kv_unified = true
build: 8091 (238856ec) with GNU 15.2.0 for Linux x86_64
system info: n_threads = 4, n_threads_batch = 4, total_threads = 8

system_info: n_threads = 4 (n_threads_batch = 4) / 8 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 

Running without SSL
init: using 7 threads for HTTP server
start: binding port with default address family
main: loading model
srv    load_model: loading model '/snap/gemma3/components/mnt/model-4b-it-q4-0-gguf/157/gemma-3-4b-it-q4_0.gguf'
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
llama_params_fit_impl: no devices with dedicated memory found
llama_params_fit: successfully fit params to free device memory
llama_params_fit: fitting params to free memory took 0.55 seconds
llama_model_loader: loaded meta data with 39 key-value pairs and 444 tensors from /snap/gemma3/components/mnt/model-4b-it-q4-0-gguf/157/gemma-3-4b-it-q4_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = gemma3
llama_model_loader: - kv   1:                      gemma3.context_length u32              = 131072
llama_model_loader: - kv   2:                         gemma3.block_count u32              = 34
llama_model_loader: - kv   3:                    gemma3.embedding_length u32              = 2560
llama_model_loader: - kv   4:                 gemma3.feed_forward_length u32              = 10240
llama_model_loader: - kv   5:                gemma3.attention.head_count u32              = 8
llama_model_loader: - kv   6:             gemma3.attention.head_count_kv u32              = 4
llama_model_loader: - kv   7:                gemma3.attention.key_length u32              = 256
llama_model_loader: - kv   8:              gemma3.attention.value_length u32              = 256
llama_model_loader: - kv   9:    gemma3.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  10:                   gemma3.rope.scaling.type str              = linear
llama_model_loader: - kv  11:                 gemma3.rope.scaling.factor f32              = 8.000000
llama_model_loader: - kv  12:                      gemma3.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  13:            gemma3.attention.sliding_window u32              = 1024
llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = llama
llama_model_loader: - kv  15:                tokenizer.ggml.bos_token_id u32              = 2
llama_model_loader: - kv  16:                tokenizer.ggml.eos_token_id u32              = 1
llama_model_loader: - kv  17:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32              = 3
llama_model_loader: - kv  19:                      tokenizer.ggml.tokens arr[str,262144]  = ["<pad>", "<eos>", "<bos>", "<unk>", ...
llama_model_loader: - kv  20:                      tokenizer.ggml.scores arr[f32,262144]  = [0.000000, 0.000000, 0.000000, 0.0000...
llama_model_loader: - kv  21:                  tokenizer.ggml.token_type arr[i32,262144]  = [3, 3, 3, 2, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  22:               general.quantization_version u32              = 2
llama_model_loader: - kv  23:                          general.file_type u32              = 2
llama_model_loader: - kv  24:                    tokenizer.chat_template str              = {{ bos_token }} {%- if messages[0]['r...
llama_model_loader: - kv  25:                 gemma3.mm.tokens_per_image u32              = 256
llama_model_loader: - kv  26:         gemma3.vision.attention.head_count u32              = 16
llama_model_loader: - kv  27: gemma3.vision.attention.layer_norm_epsilon f32              = 0.000001
llama_model_loader: - kv  28:                  gemma3.vision.block_count u32              = 27
llama_model_loader: - kv  29:             gemma3.vision.embedding_length u32              = 1152
llama_model_loader: - kv  30:          gemma3.vision.feed_forward_length u32              = 4304
llama_model_loader: - kv  31:                   gemma3.vision.image_size u32              = 896
llama_model_loader: - kv  32:                 gemma3.vision.num_channels u32              = 3
llama_model_loader: - kv  33:                   gemma3.vision.patch_size u32              = 14
llama_model_loader: - kv  34:               tokenizer.ggml.add_bos_token bool             = true
llama_model_loader: - kv  35:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  36:           tokenizer.ggml.add_padding_token bool             = false
llama_model_loader: - kv  37:           tokenizer.ggml.add_unknown_token bool             = false
llama_model_loader: - kv  38:                         tokenizer.ggml.pre str              = default
llama_model_loader: - type  f32:  205 tensors
llama_model_loader: - type  f16:    1 tensors
llama_model_loader: - type q4_0:  238 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_0
print_info: file size   = 2.93 GiB (6.49 BPW) 
load: 0 unused tokens
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load:   - 1 ('<eos>')
load:   - 106 ('<end_of_turn>')
load: special tokens cache size = 8
load: token to piece cache size = 1.9446 MB
print_info: arch                  = gemma3
print_info: vocab_only            = 0
print_info: no_alloc              = 0
print_info: n_ctx_train           = 131072
print_info: n_embd                = 2560
print_info: n_embd_inp            = 2560
print_info: n_layer               = 34
print_info: n_head                = 8
print_info: n_head_kv             = 4
print_info: n_rot                 = 256
print_info: n_swa                 = 1024
print_info: is_swa_any            = 1
print_info: n_embd_head_k         = 256
print_info: n_embd_head_v         = 256
print_info: n_gqa                 = 2
print_info: n_embd_k_gqa          = 1024
print_info: n_embd_v_gqa          = 1024
print_info: f_norm_eps            = 0.0e+00
print_info: f_norm_rms_eps        = 1.0e-06
print_info: f_clamp_kqv           = 0.0e+00
print_info: f_max_alibi_bias      = 0.0e+00
print_info: f_logit_scale         = 0.0e+00
print_info: f_attn_scale          = 6.2e-02
print_info: n_ff                  = 10240
print_info: n_expert              = 0
print_info: n_expert_used         = 0
print_info: n_expert_groups       = 0
print_info: n_group_used          = 0
print_info: causal attn           = 1
print_info: pooling type          = 0
print_info: rope type             = 2
print_info: rope scaling          = linear
print_info: freq_base_train       = 1000000.0
print_info: freq_scale_train      = 0.125
print_info: freq_base_swa         = 10000.0
print_info: freq_scale_swa        = 1
print_info: n_ctx_orig_yarn       = 131072
print_info: rope_yarn_log_mul     = 0.0000
print_info: rope_finetuned        = unknown
print_info: model type            = 4B
print_info: model params          = 3.88 B
print_info: general.name          = n/a
print_info: vocab type            = SPM
print_info: n_vocab               = 262144
print_info: n_merges              = 0
print_info: BOS token             = 2 '<bos>'
print_info: EOS token             = 1 '<eos>'
print_info: EOT token             = 106 '<end_of_turn>'
print_info: UNK token             = 3 '<unk>'
print_info: PAD token             = 0 '<pad>'
print_info: LF token              = 248 '<0x0A>'
print_info: EOG token             = 1 '<eos>'
print_info: EOG token             = 106 '<end_of_turn>'
print_info: max token length      = 93
load_tensors: loading model tensors, this can take a while... (mmap = true, direct_io = false)
load_tensors:   CPU_Mapped model buffer size =  3002.65 MiB
load_tensors:   CPU_REPACK model buffer size =  1721.25 MiB
...........................................................
common_init_result: added <eos> logit bias = -inf
common_init_result: added <end_of_turn> logit bias = -inf
llama_context: constructing llama_context
llama_context: n_seq_max     = 4
llama_context: n_ctx         = 131072
llama_context: n_ctx_seq     = 131072
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = auto
llama_context: kv_unified    = true
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 0.125
llama_context:        CPU  output buffer size =     4.00 MiB
llama_kv_cache_iswa: creating non-SWA KV cache, size = 131072 cells
llama_kv_cache:        CPU KV buffer size =  2560.00 MiB
llama_kv_cache: size = 2560.00 MiB (131072 cells,   5 layers,  4/1 seqs), K (f16): 1280.00 MiB, V (f16): 1280.00 MiB
llama_kv_cache_iswa: creating     SWA KV cache, size = 4608 cells
llama_kv_cache:        CPU KV buffer size =   522.00 MiB
llama_kv_cache: size =  522.00 MiB (  4608 cells,  29 layers,  4/1 seqs), K (f16):  261.00 MiB, V (f16):  261.00 MiB
sched_reserve: reserving ...
sched_reserve: Flash Attention was auto, set to enabled
sched_reserve:        CPU compute buffer size =   522.00 MiB
sched_reserve: graph nodes  = 1369
sched_reserve: graph splits = 1
sched_reserve: reserve took 4.59 ms, sched copies = 1
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
srv    load_model: initializing slots, n_slots = 4
no implementations specified for speculative decoding
slot   load_model: id  0 | task -1 | speculative decoding context not initialized
slot   load_model: id  0 | task -1 | new slot, n_ctx = 131072
no implementations specified for speculative decoding
slot   load_model: id  1 | task -1 | speculative decoding context not initialized
slot   load_model: id  1 | task -1 | new slot, n_ctx = 131072
no implementations specified for speculative decoding
slot   load_model: id  2 | task -1 | speculative decoding context not initialized
slot   load_model: id  2 | task -1 | new slot, n_ctx = 131072
no implementations specified for speculative decoding
slot   load_model: id  3 | task -1 | speculative decoding context not initialized
slot   load_model: id  3 | task -1 | new slot, n_ctx = 131072
srv    load_model: prompt cache is enabled, size limit: 8192 MiB
srv    load_model: use `--cache-ram 0` to disable the prompt cache
srv    load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
init: chat template, example_format: '<start_of_turn>user
You are a helpful assistant
Hello<end_of_turn>
<start_of_turn>model
Hi there<end_of_turn>
<start_of_turn>user
How are you?<end_of_turn>
<start_of_turn>model
'
srv          init: init: chat template, thinking = 0
main: model loaded
main: server is listening on http://0.0.0.0:8080
main: starting the main loop...
srv  update_slots: all slots are idle




srv  params_from_: Chat format: Content-only
slot get_availabl: id  3 | task -1 | selected slot by LRU, t_last = -1
slot launch_slot_: id  3 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> temp-ext -> dist 
slot launch_slot_: id  3 | task 0 | processing task, is_child = 0
slot update_slots: id  3 | task 0 | new prompt, n_ctx_slot = 131072, n_keep = 0, task.n_tokens = 24
slot update_slots: id  3 | task 0 | n_tokens = 0, memory_seq_rm [0, end)
slot update_slots: id  3 | task 0 | prompt processing progress, n_tokens = 24, batch.n_tokens = 24, progress = 1.000000
slot update_slots: id  3 | task 0 | prompt done, n_tokens = 24, batch.n_tokens = 24
slot init_sampler: id  3 | task 0 | init sampler, took 0.02 ms, tokens: text = 24, total = 24
slot print_timing: id  3 | task 0 | 
prompt eval time =     682.91 ms /    24 tokens (   28.45 ms per token,    35.14 tokens per second)
       eval time =   13437.19 ms /    90 tokens (  149.30 ms per token,     6.70 tokens per second)
      total time =   14120.10 ms /   114 tokens
slot      release: id  3 | task 0 | stop processing: n_tokens = 113, truncated = 0
srv  update_slots: all slots are idle
srv  log_server_r: done request: POST /v1/chat/completions 127.0.0.1 200






^Csrv    operator(): operator(): cleaning up before exit...
llama_memory_breakdown_print: | memory breakdown [MiB] | total   free    self   model   context   compute    unaccounted |
llama_memory_breakdown_print: |   - Host               |                 6606 =  3002 +    3082 +     522                |
llama_memory_breakdown_print: |   - CPU_REPACK         |                 1721 =  1721 +       0 +       0                |

[jpm-canonical]

From further investigation it seems like the problem is the order in which the TCP connection is closed when using streaming mode. The client initiates the TCP connections, but it looks like the server closes the connection. This causes the socket to go into the TIME_WAIT state - waiting for all data to be cleared out of the socket buffers. If the client instead initiated and closed the socket, the socket would not go into the TIME_WAIT state.

I have found some workarounds:

1. Add SO_REUSEADDR to the listening socket. This will allow a new instance of llama-server to immediately bind to the port, even though it is still in TIME_WAIT state. @Green-Sky mentioned here that this could introduce a port hijacking possibility.
2. Add SO_LINGER with a timeout of 0 seconds. This will cause the TCP connection to be immediately reset when the connection is closed from the server side, avoiding the TIME_WAIT state.

[jpm-canonical]

This issue was solved by #21056 which introduced the SO_REUSEADDR socket option.