server: add dynamic control vector management endpoints

[gapeleon]

This implements the ability to load, unload, and update control vectors (representation engineering) mid-inference via the server API. It follows the existing SERVER_TASK queue pattern used by LoRA adapters to ensure thread safety within the main loop.

New Endpoints:

• GET /control-vectors
• POST /control-vectors/load
• POST /control-vectors/unload
• POST /control-vectors/apply (handles scaling)

Implementation Notes:

• The application logic has been centralized in apply_control_vectors_internal to share implementation between load, unload, and apply tasks.
• Vectors are applied globally to the model context.
• Enforces dimension validation on load to gracefully reject incompatible vectors. (eg. trying to load a GLM-4.6 cvector when Devstral-2-123B is loaded)

Stability:
I have been using this implementation as my daily driver for inference steering for the last ~2-3 months.

Validation:

Verified using Devstral-2-123B. Control vectors successfully steer output semantics while maintaining syntax consistency.

Example: "Draw an SVG of a cute cat"

Baseline: 

Nihilism and Machiavellianism Vector (+1.0): 

[saood06]

Thanks for making this PR. I remember being interested in ggml-org/llama.cpp#6289 way back when, and dissapointed it went nowhere.

[jukofyork]

Verified using Devstral-2-123B. Control vectors successfully steer output semantics while maintaining syntax consistency.

Example: "Draw an SVG of a cute cat"

Baseline: 

Nihilism and Machiavellianism Vector (+1.0): 

LOL, I never thought of using these for generating images! 😂

[gapeleon]

@saood06 Thanks, I'm glad to hear there's interest! I hadn't seen that previous PR.
It looks like they were trying to call llama_control_vector_apply(ctx_server.ctx, ...) directly inside the HTTP handler, hence ran into memory safety and race conditions during the hot-swap.

This implementation is modelled after the current LoRA adapter slot persistence and server_task queue architecture, which sidesteps those issues.
All the vector (un)loading and scaling happens within the main event loop.

@jukofyork Haha glad you liked the cat test! It's arguably the rigorous proof that the vectors are shifting semantics without breaking syntax, and I figured it demonstrates the feature at a glance rather than asking everyone to read 2000 word short stories. It also works for css/website mock-ups.

If anyone wants to test the endpoints quickly, here are the curl snippets and some vectors I used:

Devstral 123B
GLM-4.5-air
@jukofyork's collection has over 70+ models (Cohere, Mistral-Large, Nemo) jukofyork/creative-writing-control-vectors-v3.0

# Load
curl http://127.0.0.1:8080/control-vectors/load \
  -X POST \
  -d '{"path": "/path/to/vector.gguf", "scale": 1.0}'

# Scale (Instant)
curl http://127.0.0.1:8080/control-vectors/apply \
  -X POST \
  -d '[{"id": 0, "scale": 1.5}]'

[ikawrakow]

Thank you for the PR.

I wasn't aware that people are actually using control vectors with ik_llama.cpp. Hence, they are not taken seriously for graph parallel, so graph parallel performance with control vectors is likely lower than it could be.