Served-backend (llama_server) engine: capability-parity follow-ups

[ttupper92618]

Summary

The served-backend engine (llama_server, added in #419) covers the core path — placement routing, native MTP (--spec-type draft-mtp), streaming, reasoning split, tool calls, cancellation, teardown — and is proven live on kite4 (2.19x). A few capability-parity items were deliberately deferred from #419 to keep it scoped; tracking them here.

Items

1. Forward thinking / reasoning-effort controls to llama-server. A thinking-toggle card (e.g. Qwen3.6) plus task_params.enable_thinking / reasoning_effort from the API adapters are not currently forwarded in the served request body (only sampling kwargs + messages are). Map them onto the right llama-server fields (e.g. chat_template_kwargs.enable_thinking, reasoning-effort/budget) and validate on kite4 that toggling thinking off is honored. (Codex P2 on feat(engine): served-backend llama_server engine for native MTP #419.)

2. Logprobs over the proxy. The served runner currently fails loud on a logprobs request (matching the llama.cpp engine returns no per-token logprobs in the streaming path (all GGUF models) #385 no-silent-empty contract). Parse the server's logprob payload and surface it on the emitted TokenChunks so served models reach logprobs parity with the in-process runner.

3. CPU-served local fit guard. When a node resolves to llama_server-cpu, the runner correctly starts with -ngl 0 and the master admits against system RAM, but the worker's pre-spawn/load fit guard (footprint_exceeds_usable) may still size that placement against local VRAM. Make the local guard aware of a served -cpu resolution so it sizes against RAM, matching the master. Narrow edge (served on CPU is uncommon; served is primarily a GPU/MTP feature). (Codex P2 on feat(engine): served-backend llama_server engine for native MTP #419.)

References

• Engine + the deferral rationale: PR feat(engine): served-backend llama_server engine for native MTP #419 (src/skulk/worker/runner/llama_server/runner.py).

[ttupper92618]

Add a 4th item (Codex P2 on #419): cancellation during a prefill stall. The served runner's streaming loop blocks in resp.iter_lines() with no read timeout, so if a request is cancelled while llama-server is still prefilling a long prompt (before the first SSE line), _is_cancelled() is not reached and the connection is not closed, so the server keeps generating/holding VRAM until it emits a line or finishes. Normal mid-stream cancellation works; this is the prefill-stall edge. Fix: a bounded read timeout + poll, or a separate control path, so cancellation can abort during prefill.

[ttupper92618]

Two findings (Gemma 4 MTP + GLM-4.7 root cause)

Gemma 4 MTP needs draft-model plumbing in the served runner

Gemma 4 does NOT bake MTP heads into the base GGUF. llama.cpp (PR ggml-org/llama.cpp#23398) added Gemma 4 MTP by loading the Gemma 4 assistant as a SEPARATE draft GGUF: llama-server -m gemma-4-31B.gguf --model-draft mtp-gemma-4-31B-it.gguf --spec-type draft-mtp --spec-draft-n-max 4. Our served runner currently passes --spec-type only (works for baked-in-head families like Qwen3.6). To support Gemma 4 (and the generic draft-simple vanilla-draft path) we need:

• a card field for the draft GGUF (repo + file), e.g. served_spec_draft_repo/served_spec_draft_file;
• the served runner to resolve + co-locate (download as a companion) the draft GGUF and pass --model-draft <path>;
• a Gemma 4 served card (base GGUF + the assistant MTP draft GGUF), served_spec_type=draft_mtp.
  This is the single missing capability; it unlocks Gemma 4 MTP AND classic draft-simple.

GLM-4.7-Flash MTP load failure = bad third-party conversion (not a limitation)

The base zai-org/GLM-4.7-Flash (Glm4MoeLiteForCausalLM) DOES ship an MTP head (config num_nextn_predict_layers: 1). llama.cpp's GLM-MTP arch expects 868 tensors incl. the 6 nextn/MTP tensors; the jamesdumay GGUF had only 862 -> the conversion DROPPED the MTP head. llama.cpp correctly rejected it (strict tensor-count check). A correct GGUF is producible.

Lesson (own the artifact, the sidecar philosophy): baked-in-head families (Qwen/DeepSeek/GLM) give llama.cpp NO sidecar slot for the heads, so we depend on the GGUF conversion being correct; a third party can silently ship a 'MTP' GGUF with the heads missing. Mitigations:

1. Validate MTP tensors are present before carding/serving a draft-mtp GGUF (a nextn/mtp tensor-count check at card creation or a fail-loud at served load) so a broken upload is caught early, not at serve time.
2. Produce the MTP GGUFs ourselves (SWP via convert_hf_to_gguf.py) for the families we rely on, guaranteeing the heads -- same principle as our MLX mtp.safetensors sidecars.
3. Gemma 4's MTP draft IS a literal sidecar (separate --model-draft GGUF) -> the case where we publish/control the assistant GGUF directly via SWP.