Skip to content

feat: add EAGLE-3 speculative decoding support for Ornith-9B #11

Description

@oniwakaa

Description

EAGLE-3 speculative decoding merged into llama.cpp in June 2026 (PR #18039). It achieves 2-3× speedup using a tiny draft head (1 transformer layer) trained on the target model's hidden states.

LORE previously skipped speculative decoding because Falcon-H1 and Ornith have incompatible vocabs. EAGLE-3 doesn't use a standalone draft model — it shares the target's tokenizer.

Task

  1. Check if AngelSlim/Qwen3-8B_eagle3 (or similar Qwen3 EAGLE-3 checkpoint) works on Ornith's Qwen3.5 architecture
  2. Convert the checkpoint to GGUF using convert_hf_to_gguf.py --target-model-dir
  3. Test with: llama-server -m ornith.gguf -md eagle3.gguf --spec-type draft-eagle3 --spec-draft-n-max 8
  4. Measure: speedup vs baseline, acceptance rate, any quality regression

Expected Impact

If it works: 2-3× faster primary model inference. This would be transformative for LORE's orchestration latency.

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions