Feature Request: Support for BitNet b1.58 2B4T model architecture

[vlasky]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

I downloaded the BitNet b1.58 2B4T GGUF file from here:

https://huggingface.co/microsoft/bitnet-b1.58-2B-4T-gguf

I am using llamafile 0.9.2.

Running llamafile.exe -m ggml-model-i2_s.gguf results in the following output:

llama.cpp/ggml.c:19663: GGML_ASSERT(0 <= info->type && info->type < GGML_TYPE_COUNT) failed

error: Uncaught SIGABRT (SI_TKILL) at 0 on MOREFINE pid 44108 tid 44156
  llamafile.exe
  No error information
  Windows Cosmopolitan 3.9.7 MODE=x86_64 MOREFINE 10.0

RAX 0000000000000000 RBX 0000000000000006 RDI 00007000007dcc10
RCX ffffffffffffffdf RDX 0000000000000000 RSI 00000000fffffffa
RBP 00007000007dcf60 RSP 00007000007dcaf0 RIP 0000000000411362
 R8 00007000007dcca8  R9 00007000007dcd00 R10 0000000000000000
R11 0000000000000246 R12 0000000000a0d45f R13 0000000000004ccf
R14 00000000007f224f R15 0000014cfba8b4d0
TLS 0000000000c44f40

XMM0  00000000000000000000000000000000 XMM8  00000000000000000000000000000000
XMM1  00000000000000000000000000000000 XMM9  00000000000000000000000000000000
XMM2  00000000000000000000000000000000 XMM10 00000000000000000000000000000000
XMM3  00000000000000000000000000000000 XMM11 00000000000000000000000000000000
XMM4  00000000000000000000000000000000 XMM12 00000000000000000000000000000000
XMM5  00000000000000000000000000000000 XMM13 00000000000000000000000000000000
XMM6  a2c3afc4b8c4a2c3afc4b8c4a2c3afc4 XMM14 00000000000000000000000000000000
XMM7  b8c4a2c3afc4b8c4a2c3afc4b8c4a2c3 XMM15 00000000000000000000000000000000

cosmoaddr2line /C/Users/Vlad/Downloads/llamafile.exe 411362 990e3e 4078c5 53197f 5577ed 5c5a0b 5b2a99 5210cf 41d7ee 404284 99bd3b

7000007da5f0 41135d __sig_raise+45
7000007dcf60 990e3e raise+78
7000007dcf80 4078c5 abort+40
7000007dcfa0 53197f ggml_abort+223
7000007dd0a0 5577ed gguf_init_from_file+3805
7000007dd170 5c5a0b llama_model_loader::llama_model_loader(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, bool, bool, llama_model_kv_override const*)+731
7000007ddaf0 5b2a99 llama_load_model_from_file+1721
7000007ddce0 5210cf lf::chatbot::main(int, char**)+319
7000007dde50 41d7ee main+814
7000007deec0 404284 cosmo+68
7000007deed0 99bd3b __stack_call+16

0000002d0000-0000002e0000 rw-Pa 64kb hand=324
0000002e0000-0000002f0000 rw-Pa 64kb
000000400000-000000b7f1e0 r-x-- 7676kb
000000b80000-0000032ff000 rw--- 39mb
0006fe000000-0006fe010000 rw-pa 64kb hand=328
014ce7e20000-014ce7e30000 rw-pa 64kb hand=332
014ce7e30000-014ce7e40000 rw-pa 64kb hand=336
014ce7e40000-014ce7e50000 rw-pa 64kb hand=340
014ce7e50000-014ce7e60000 rw-pa 64kb hand=344
014ce7e60000-014ce7e70000 rw-pa 64kb hand=348
014ce7e70000-014ce7e80000 rw-pa 64kb hand=352
014ce7e80000-014ce7e90000 rw-pa 64kb hand=356
014ce7e90000-014ce7ea0000 rw-pa 64kb hand=360
014ce7ea0000-014ce7eb0000 rw-pa 64kb hand=364
014ce7eb0000-014ce7ec0000 rw-pa 64kb hand=384
# 390'832'128 bytes in 34 mappings

  0 kFdConsole handle=84
  1 kFdConsole flags=O_WRONLY|O_APPEND handle=88
  2 kFdConsole flags=O_WRONLY|O_APPEND handle=92
  3 kFdFile flags=O_RDONLY|O_CLOEXEC handle=428

Motivation

Microsoft's BitNet b1.58 2B4T sounds like an interesting model architecture. From the Hugging Face page:

"Trained on a corpus of 4 trillion tokens, this model demonstrates that native 1-bit LLMs can achieve performance comparable to leading open-weight, full-precision models of similar size, while offering substantial advantages in computational efficiency (memory, energy, latency)."

Possible Implementation

No response

[vlasky]

If it helps, ik_llama.cpp (llama.cpp fork) has support for it:

ikawrakow/ik_llama.cpp#337

[cjpais]

If you want to help port the code I will be happy to test and pull it in