Description
EAGLE-3 speculative decoding merged into llama.cpp in June 2026 (PR #18039). It achieves 2-3× speedup using a tiny draft head (1 transformer layer) trained on the target model's hidden states.
LORE previously skipped speculative decoding because Falcon-H1 and Ornith have incompatible vocabs. EAGLE-3 doesn't use a standalone draft model — it shares the target's tokenizer.
Task
- Check if
AngelSlim/Qwen3-8B_eagle3 (or similar Qwen3 EAGLE-3 checkpoint) works on Ornith's Qwen3.5 architecture
- Convert the checkpoint to GGUF using
convert_hf_to_gguf.py --target-model-dir
- Test with:
llama-server -m ornith.gguf -md eagle3.gguf --spec-type draft-eagle3 --spec-draft-n-max 8
- Measure: speedup vs baseline, acceptance rate, any quality regression
Expected Impact
If it works: 2-3× faster primary model inference. This would be transformative for LORE's orchestration latency.
References
Description
EAGLE-3 speculative decoding merged into llama.cpp in June 2026 (PR #18039). It achieves 2-3× speedup using a tiny draft head (1 transformer layer) trained on the target model's hidden states.
LORE previously skipped speculative decoding because Falcon-H1 and Ornith have incompatible vocabs. EAGLE-3 doesn't use a standalone draft model — it shares the target's tokenizer.
Task
AngelSlim/Qwen3-8B_eagle3(or similar Qwen3 EAGLE-3 checkpoint) works on Ornith's Qwen3.5 architectureconvert_hf_to_gguf.py --target-model-dirllama-server -m ornith.gguf -md eagle3.gguf --spec-type draft-eagle3 --spec-draft-n-max 8Expected Impact
If it works: 2-3× faster primary model inference. This would be transformative for LORE's orchestration latency.
References