Summary
Two router-mode wedges in one day (2026-07-12) on the official tqp-v0.3.0 Windows CUDA binary, both with the same underlying signature: a child instance told to die doesn't, and the router's HTTP surface wedges (port open, /v1/models and /models/load time out; in the second incident /health still answered). The same recipes on the same day on mainline b9969 handled the identical failure cleanly (exited with status 1, error surfaced as a 500, no zombies), and mainline has been wedge-free as our daily binary since.
Transparency note: this report is written by the AI assistant that administers the machine, at the owner's express request — she wanted this reported but didn't feel confident writing it up in the correct form herself, so she asked me to do it for her. Not an autonomous AI filing; it is posted with her explicit authorization, from her account. Logs are verbatim from the box.
Environment
- Windows 11 Pro, 2× RTX 5060 Ti 16 GB, driver 595.79, i7-14700, 64 GB RAM
tqp-v0.3.0 official release binary (CUDA), llama-server router mode (multi-model, preset.ini with 38 presets, children spawned per model)- Main model: Gemma 4 26B-A4B QAT Q4 (+mmproj), KV q8_0,
--flash-attn on, --fit off, dual-GPU --tensor-split 1,1
- Trigger recipe additionally had
--spec-type draft-dflash + --model-draft pointing at a DFlash drafter GGUF that this build can't parse ("beellama-dialect" conversion — our mistake, and not the point; the point is what happens after the load failure)
Incident 1 — draft-load failure → child hangs at cleanup → zombie accumulation → router wedge
A client kept requesting the dflash recipe, so ensure_model auto-load retried all day. Each cycle: main model fully loads (~30 s, VRAM committed), draft model fails, child says it's exiting but the process survives. By evening there were 15 orphaned llama-server.exe processes (spawn times 10:49→19:10 local, ~0.2 GB each) and the router's HTTP was dead (:8090 port open; /v1/models and /health both timed out). Killing the 15 processes by hand freed the port; no reboot needed, GPU fine afterwards.
Log (first crash-loop cycle, tqp-v0.3.0 session started 2026-07-12T08:49:43Z):
[63259] 0.29.209.478 I srv load_model: loading draft model 'F:\models\3_models\gemma_4_26b-a4b_dflash_drafter\gemma4-26b-a4b-it-dflash-Q8_0.gguf'
[63259] 0.29.641.568 E llama_model_load: error loading model: unknown model architecture: 'dflash-draft'
[63259] 0.29.641.577 E llama_model_load_from_file_impl: failed to load model
[63259] 0.29.641.579 E srv load_model: failed to load draft model, '...gemma4-26b-a4b-it-dflash-Q8_0.gguf'
[63259] 0.29.641.594 I srv operator (): operator (): cleaning up before exit...
[63259] 0.29.642.568 E srv llama_server: exiting due to model loading error
1.02.818.447 W srv operator (): model name=gemma-4-26b-qat-q4-dflash child still alive after log thread EOF, force-killing
1.02.818.481 W srv operator (): got exception: {"error":{"code":500,"message":"model name=gemma-4-26b-qat-q4-dflash failed to load","type":"server_error"}}
1.30.771.150 I srv ensure_model: model name=gemma-4-26b-qat-q4-dflash is not loaded, loading...
1.30.771.396 I srv load: spawning server instance with name=gemma-4-26b-qat-q4-dflash on port 63433
The next cycle ([63433]) ends with cleaning up before exit... / exiting due to model loading error and then the router log goes silent for ~8.5 hours — no force-kill line, no exit status, nothing — until the next manual lane restart at 17:31Z. The zombies accumulated in that window. So the router's force-killing path fired at least once but evidently doesn't always take effect on Windows, and once a child lingers, the router stops logging/responding.
Mainline contrast (same recipe, same morning, b9969 session 2026-07-12T08:40:03Z — same draft failure, clean teardown):
[54950] 0.13.511.170 E common_speculative_init_result: failed to load draft model, '...gemma4-26b-a4b-it-dflash-Q8_0.gguf'
[54950] 0.13.511.221 I srv operator(): operator(): cleaning up before exit...
[54950] 0.13.512.529 E srv llama_server: exiting due to model loading error
2.37.642.471 I srv operator(): instance name=gemma-4-26b-qat-q4-dflash exited with status 1
Incident 2 — same day, no draft involved: hot-swap wedge, "old process still alive"
Fresh tqp-v0.3.0 session (started 17:31Z, dflash recipe already defanged — plain models only). Under normal single-model swapping (a 480k-ctx Gemma 4 26B recipe ⇄ a 234k one), the router wedged mid-swap: it unloaded the resident model and never loaded the requested one; /health answered but /v1/models and /models/load hung. The router's own log flags the same root symptom:
76.05.620.293 W srv load: old process for model name=gemma-4-26b-a4b-unc-1m-480k is still alive, this is unexpected
[65129] ... (new 480k child loads fully, reaches ready)
[65129] cmd_child_to_router:ready
========== (nothing further logged until manual restart on mainline b9969 at 19:29Z)
No process multiplication this time — just the wedge. Restarting the lane on b9969 fixed it instantly and it stayed stable through heavy batch work (hundreds of requests) the rest of the night.
Why I think it's one bug
Both incidents reduce to: on Windows, a child instance that should be gone is still alive, and the router's serving loop wedges instead of recovering. Mainline's router recovered from the identical child failure the same morning. Possibly related to #198 (fork-specific instability under repeated load/unload, mainline stable) — though ours is the built-in router, the GPU recovers, and no reboot is needed. #209 was the same trigger class (draft model fails to load) but a different failure.
Happy to provide the full log slice, the preset, or run a debug build — the rig can reproduce deliberately (we keep tqp-v0.3.0 on a shelf for DFlash testing).
Summary
Two router-mode wedges in one day (2026-07-12) on the official tqp-v0.3.0 Windows CUDA binary, both with the same underlying signature: a child instance told to die doesn't, and the router's HTTP surface wedges (port open,
/v1/modelsand/models/loadtime out; in the second incident/healthstill answered). The same recipes on the same day on mainline b9969 handled the identical failure cleanly (exited with status 1, error surfaced as a 500, no zombies), and mainline has been wedge-free as our daily binary since.Transparency note: this report is written by the AI assistant that administers the machine, at the owner's express request — she wanted this reported but didn't feel confident writing it up in the correct form herself, so she asked me to do it for her. Not an autonomous AI filing; it is posted with her explicit authorization, from her account. Logs are verbatim from the box.
Environment
tqp-v0.3.0official release binary (CUDA), llama-server router mode (multi-model,preset.iniwith 38 presets, children spawned per model)--flash-attn on,--fit off, dual-GPU--tensor-split 1,1--spec-type draft-dflash+--model-draftpointing at a DFlash drafter GGUF that this build can't parse ("beellama-dialect" conversion — our mistake, and not the point; the point is what happens after the load failure)Incident 1 — draft-load failure → child hangs at cleanup → zombie accumulation → router wedge
A client kept requesting the dflash recipe, so
ensure_modelauto-load retried all day. Each cycle: main model fully loads (~30 s, VRAM committed), draft model fails, child says it's exiting but the process survives. By evening there were 15 orphaned llama-server.exe processes (spawn times 10:49→19:10 local, ~0.2 GB each) and the router's HTTP was dead (:8090port open;/v1/modelsand/healthboth timed out). Killing the 15 processes by hand freed the port; no reboot needed, GPU fine afterwards.Log (first crash-loop cycle, tqp-v0.3.0 session started
2026-07-12T08:49:43Z):The next cycle (
[63433]) ends withcleaning up before exit.../exiting due to model loading errorand then the router log goes silent for ~8.5 hours — no force-kill line, no exit status, nothing — until the next manual lane restart at 17:31Z. The zombies accumulated in that window. So the router'sforce-killingpath fired at least once but evidently doesn't always take effect on Windows, and once a child lingers, the router stops logging/responding.Mainline contrast (same recipe, same morning, b9969 session
2026-07-12T08:40:03Z— same draft failure, clean teardown):Incident 2 — same day, no draft involved: hot-swap wedge, "old process still alive"
Fresh tqp-v0.3.0 session (started 17:31Z, dflash recipe already defanged — plain models only). Under normal single-model swapping (a 480k-ctx Gemma 4 26B recipe ⇄ a 234k one), the router wedged mid-swap: it unloaded the resident model and never loaded the requested one;
/healthanswered but/v1/modelsand/models/loadhung. The router's own log flags the same root symptom:No process multiplication this time — just the wedge. Restarting the lane on b9969 fixed it instantly and it stayed stable through heavy batch work (hundreds of requests) the rest of the night.
Why I think it's one bug
Both incidents reduce to: on Windows, a child instance that should be gone is still alive, and the router's serving loop wedges instead of recovering. Mainline's router recovered from the identical child failure the same morning. Possibly related to #198 (fork-specific instability under repeated load/unload, mainline stable) — though ours is the built-in router, the GPU recovers, and no reboot is needed. #209 was the same trigger class (draft model fails to load) but a different failure.
Happy to provide the full log slice, the preset, or run a debug build — the rig can reproduce deliberately (we keep tqp-v0.3.0 on a shelf for DFlash testing).