Skip to content

Feature implementation from commits 933e895..874e6bd #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
yashuatla wants to merge 16 commits into
base: feature-base-3
Choose a base branch
from
Open

Feature implementation from commits 933e895..874e6bd #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
22566ab
Make the coverage table more readable with both dark and light theme
lfoppiano Dec 5, 2024
2539548
Merge pull request #130 from lfoppiano/patch-1
potassiummmm Dec 20, 2024
1791e8e
Merge branch 'microsoft:main' into readme-new-model
potassiummmm Dec 20, 2024
d0fc8c9
Update README.md
potassiummmm Dec 20, 2024
437b321
Merge pull request #145 from potassiummmm/readme-new-model
potassiummmm Dec 20, 2024
caf17ec
update README
Eddie-Wang1120 Feb 18, 2025
4f2e41a
add support for bitnet2b_2501 model
potassiummmm Mar 12, 2025
09f9106
add conversion logic for new model
potassiummmm Mar 12, 2025
fa854cf
Merge pull request #167 from potassiummmm/bitnet-25
tsong-ms Apr 15, 2025
fd3f355
update readme and setup script to support official BitNet b1.58 model…
sd983527 Apr 15, 2025
0e7dadb
Update README.md
sd983527 Apr 15, 2025
8f75f99
Update README.md (#172)
sd983527 Apr 15, 2025
1c77bd8
Update README.md
sd983527 Apr 15, 2025
71fdd94
add third-party demo
tsong-ms Apr 15, 2025
034b34c
Merge pull request #175 from microsoft/readme-dev
tsong-ms Apr 15, 2025
874e6bd
refine readme
tsong-ms Apr 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion 3rdparty/llama.cpp
Open in desktop
Submodule llama.cpp updated 3 files
+24 −0 gguf-py/gguf/constants.py
+5 −0 gguf-py/gguf/tensor_mapping.py
+298 −1 src/llama.cpp
215 changes: 97 additions & 118 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@
[![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://opensource.org/licenses/MIT)
![version](https://img.shields.io/badge/version-1.0-blue)

[<img src="./assets/header_model_release.png" alt="BitNet Model on Hugging Face" width="800"/>](https://huggingface.co/microsoft/BitNet-b1.58-2B-4T)

Try it out via this [demo](https://bitnet-demo.azurewebsites.net/), or [build and run](https://github.com/microsoft/BitNet?tab=readme-ov-file#build-from-source) it on your own CPU.

bitnet.cpp is the official inference framework for 1-bit LLMs (e.g., BitNet b1.58). It offers a suite of optimized kernels, that support **fast** and **lossless** inference of 1.58-bit models on CPU (with NPU and GPU support coming next).

The first release of bitnet.cpp is to support inference on CPUs. bitnet.cpp achieves speedups of **1.37x** to **5.07x** on ARM CPUs, with larger models experiencing greater performance gains. Additionally, it reduces energy consumption by **55.4%** to **70.0%**, further boosting overall efficiency. On x86 CPUs, speedups range from **2.37x** to **6.17x** with energy reductions between **71.9%** to **82.2%**. Furthermore, bitnet.cpp can run a 100B BitNet b1.58 model on a single CPU, achieving speeds comparable to human reading (5-7 tokens per second), significantly enhancing the potential for running LLMs on local devices. Please refer to the [technical report](https://arxiv.org/abs/2410.16144) for more details.
Expand All @@ -18,8 +22,9 @@ A demo of bitnet.cpp running a BitNet b1.58 3B model on Apple M2:
https://github.com/user-attachments/assets/7f46b736-edec-4828-b809-4be780a3e5b1

## What's New:

- 11/08/2024 [BitNet a4.8: 4-bit Activations for 1-bit LLMs](https://arxiv.org/abs/2411.04965) ![NEW](https://img.shields.io/badge/NEW-red)
- 04/14/2025 [BitNet Official 2B Parameter Model on Hugging Face](https://huggingface.co/microsoft/BitNet-b1.58-2B-4T) ![NEW](https://img.shields.io/badge/NEW-red)
- 02/18/2025 [Bitnet.cpp: Efficient Edge Inference for Ternary LLMs](https://arxiv.org/abs/2502.11880)
- 11/08/2024 [BitNet a4.8: 4-bit Activations for 1-bit LLMs](https://arxiv.org/abs/2411.04965)
- 10/21/2024 [1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs](https://arxiv.org/abs/2410.16144)
- 10/17/2024 bitnet.cpp 1.0 released.
- 03/21/2024 [The-Era-of-1-bit-LLMs__Training_Tips_Code_FAQ](https://github.com/microsoft/unilm/blob/master/bitnet/The-Era-of-1-bit-LLMs__Training_Tips_Code_FAQ.pdf)
Expand All @@ -29,10 +34,7 @@ https://github.com/user-attachments/assets/7f46b736-edec-4828-b809-4be780a3e5b1
## Acknowledgements

This project is based on the [llama.cpp](https://github.com/ggerganov/llama.cpp) framework. We would like to thank all the authors for their contributions to the open-source community. Also, bitnet.cpp's kernels are built on top of the Lookup Table methodologies pioneered in [T-MAC](https://github.com/microsoft/T-MAC/). For inference of general low-bit LLMs beyond ternary models, we recommend using T-MAC.

## Supported Models
❗️**We use existing 1-bit LLMs available on [Hugging Face](https://huggingface.co/) to demonstrate the inference capabilities of bitnet.cpp. These models are neither trained nor released by Microsoft. We hope the release of bitnet.cpp will inspire the development of 1-bit LLMs in large-scale settings in terms of model size and training tokens.**

## Official Models
<table>
</tr>
<tr>
Expand All @@ -47,144 +49,92 @@ This project is based on the [llama.cpp](https://github.com/ggerganov/llama.cpp)
<th>TL2</th>
</tr>
<tr>
<td rowspan="2"><a href="https://huggingface.co/1bitLLM/bitnet_b1_58-large">bitnet_b1_58-large</a></td>
<td rowspan="2">0.7B</td>
<td>x86</td>
<td>&#10004;</td>
<td>&#10008;</td>
<td>&#10004;</td>
</tr>
<tr>
<td>ARM</td>
<td>&#10004;</td>
<td>&#10004;</td>
<td>&#10008;</td>
</tr>
<tr>
<td rowspan="2"><a href="https://huggingface.co/1bitLLM/bitnet_b1_58-3B">bitnet_b1_58-3B</a></td>
<td rowspan="2">3.3B</td>
<td rowspan="2"><a href="https://huggingface.co/microsoft/BitNet-b1.58-2B-4T">BitNet-b1.58-2B-4T</a></td>
<td rowspan="2">2.4B</td>
<td>x86</td>
<td>&#10008;</td>
<td>&#10008;</td>
<td>&#10004;</td>
<td>&#9989;</td>
<td>&#10060;</td>
<td>&#9989;</td>
</tr>
<tr>
<td>ARM</td>
<td>&#10008;</td>
<td>&#10004;</td>
<td>&#10008;</td>
<td>&#9989;</td>
<td>&#9989;</td>
<td>&#10060;</td>
</tr>
<tr>
<td rowspan="2"><a href="https://huggingface.co/HF1BitLLM/Llama3-8B-1.58-100B-tokens">Llama3-8B-1.58-100B-tokens</a></td>
<td rowspan="2">8.0B</td>
<td>x86</td>
<td>&#10004;</td>
<td>&#10008;</td>
<td>&#10004;</td>
</tr>
<tr>
<td>ARM</td>
<td>&#10004;</td>
<td>&#10004;</td>
<td>&#10008;</td>
</tr>
<tr>
<td rowspan="2"><a href="https://huggingface.co/tiiuae/Falcon3-1B-Instruct-1.58bit">Falcon3-1B-Instruct-1.58bit</a></td>
<td rowspan="2">1.0B</td>
<td>x86</td>
<td>&#10004;</td>
<td>&#10008;</td>
<td>&#10004;</td>
</tr>
<tr>
<td>ARM</td>
<td>&#10004;</td>
<td>&#10004;</td>
<td>&#10008;</td>
</tr>
<tr>
<td rowspan="2"><a href="https://huggingface.co/tiiuae/Falcon3-3B-1.58bit">Falcon3-3B-1.58bit</a></td>
<td rowspan="2">3.0B</td>
<td>x86</td>
<td>&#10004;</td>
<td>&#10008;</td>
<td>&#10004;</td>
</tr>
<tr>
<td>ARM</td>
<td>&#10004;</td>
<td>&#10004;</td>
<td>&#10008;</td>
</table>

## Supported Models
❗️**We use existing 1-bit LLMs available on [Hugging Face](https://huggingface.co/) to demonstrate the inference capabilities of bitnet.cpp. We hope the release of bitnet.cpp will inspire the development of 1-bit LLMs in large-scale settings in terms of model size and training tokens.**

<table>
</tr>
<tr>
<td rowspan="2"><a href="https://huggingface.co/tiiuae/Falcon3-3B-Instruct-1.58bit">Falcon3-3B-Instruct-1.58bit</a></td>
<td rowspan="2">3.0B</td>
<td>x86</td>
<td>&#10004;</td>
<td>&#10008;</td>
<td>&#10004;</td>
<th rowspan="2">Model</th>
<th rowspan="2">Parameters</th>
<th rowspan="2">CPU</th>
<th colspan="3">Kernel</th>
</tr>
<tr>
<td>ARM</td>
<td>&#10004;</td>
<td>&#10004;</td>
<td>&#10008;</td>
<th>I2_S</th>
<th>TL1</th>
<th>TL2</th>
</tr>
<tr>
<td rowspan="2"><a href="https://huggingface.co/tiiuae/Falcon3-7B-1.58bit">Falcon3-7B-1.58bit</a></td>
<td rowspan="2">7.0B</td>
<td rowspan="2"><a href="https://huggingface.co/1bitLLM/bitnet_b1_58-large">bitnet_b1_58-large</a></td>
<td rowspan="2">0.7B</td>
<td>x86</td>
<td>&#10004;</td>
<td>&#10008;</td>
<td>&#10004;</td>
<td>&#9989;</td>
<td>&#10060;</td>
<td>&#9989;</td>
</tr>
<tr>
<td>ARM</td>
<td>&#10004;</td>
<td>&#10004;</td>
<td>&#10008;</td>
<td>&#9989;</td>
<td>&#9989;</td>
<td>&#10060;</td>
</tr>
<tr>
<td rowspan="2"><a href="https://huggingface.co/tiiuae/Falcon3-7B-Instruct-1.58bit">Falcon3-7B-Instruct-1.58bit</a></td>
<td rowspan="2">7.0B</td>
<td rowspan="2"><a href="https://huggingface.co/1bitLLM/bitnet_b1_58-3B">bitnet_b1_58-3B</a></td>
<td rowspan="2">3.3B</td>
<td>x86</td>
<td>&#10004;</td>
<td>&#10008;</td>
<td>&#10004;</td>
<td>&#10060;</td>
<td>&#10060;</td>
<td>&#9989;</td>
</tr>
<tr>
<td>ARM</td>
<td>&#10004;</td>
<td>&#10004;</td>
<td>&#10008;</td>
<td>&#10060;</td>
<td>&#9989;</td>
<td>&#10060;</td>
</tr>
<tr>
<td rowspan="2"><a href="https://huggingface.co/tiiuae/Falcon3-10B-1.58bit">Falcon3-10B-1.58bit</a></td>
<td rowspan="2">10.0B</td>
<td rowspan="2"><a href="https://huggingface.co/HF1BitLLM/Llama3-8B-1.58-100B-tokens">Llama3-8B-1.58-100B-tokens</a></td>
<td rowspan="2">8.0B</td>
<td>x86</td>
<td>&#10004;</td>
<td>&#10008;</td>
<td>&#10004;</td>
<td>&#9989;</td>
<td>&#10060;</td>
<td>&#9989;</td>
</tr>
<tr>
<td>ARM</td>
<td>&#10004;</td>
<td>&#10004;</td>
<td>&#10008;</td>
<td>&#9989;</td>
<td>&#9989;</td>
<td>&#10060;</td>
</tr>
<tr>
<td rowspan="2"><a href="https://huggingface.co/tiiuae/Falcon3-10B-Instruct-1.58bit">Falcon3-10B-Instruct-1.58bit</a></td>
<td rowspan="2">10.0B</td>
<td rowspan="2"><a href="https://huggingface.co/collections/tiiuae/falcon3-67605ae03578be86e4e87026">Falcon3 Family</a></td>
<td rowspan="2">1B-10B</td>
<td>x86</td>
<td>&#10004;</td>
<td>&#10008;</td>
<td>&#10004;</td>
<td>&#9989;</td>
<td>&#10060;</td>
<td>&#9989;</td>
</tr>
<tr>
<td>ARM</td>
<td>&#10004;</td>
<td>&#10004;</td>
<td>&#10008;</td>
<td>&#9989;</td>
<td>&#9989;</td>
<td>&#10060;</td>
</tr>
</table>

Expand All @@ -210,7 +160,7 @@ This project is based on the [llama.cpp](https://github.com/ggerganov/llama.cpp)
### Build from source

> [!IMPORTANT]
> If you are using Windows, please remember to always use a Developer Command Prompt / PowerShell for VS2022 for the following commands
> If you are using Windows, please remember to always use a Developer Command Prompt / PowerShell for VS2022 for the following commands. Please refer to the FAQs below if you see any issues.

1. Clone the repo
```bash
Expand All @@ -227,12 +177,10 @@ pip install -r requirements.txt
```
3. Build the project
```bash
# Download the model from Hugging Face, convert it to quantized gguf format, and build the project
python setup_env.py --hf-repo tiiuae/Falcon3-7B-Instruct-1.58bit -q i2_s
# Manually download the model and run with local path
huggingface-cli download microsoft/BitNet-b1.58-2B-4T-gguf --local-dir models/BitNet-b1.58-2B-4T
python setup_env.py -md models/BitNet-b1.58-2B-4T -q i2_s

# Or you can manually download the model and run with local path
huggingface-cli download tiiuae/Falcon3-7B-Instruct-1.58bit --local-dir models/Falcon3-7B-Instruct-1.58bit
python setup_env.py -md models/Falcon3-7B-Instruct-1.58bit -q i2_s
```
<pre>
usage: setup_env.py [-h] [--hf-repo {1bitLLM/bitnet_b1_58-large,1bitLLM/bitnet_b1_58-3B,HF1BitLLM/Llama3-8B-1.58-100B-tokens,tiiuae/Falcon3-1B-Instruct-1.58bit,tiiuae/Falcon3-3B-Instruct-1.58bit,tiiuae/Falcon3-7B-Instruct-1.58bit,tiiuae/Falcon3-10B-Instruct-1.58bit}] [--model-dir MODEL_DIR] [--log-dir LOG_DIR] [--quant-type {i2_s,tl1}] [--quant-embd]
Expand All @@ -257,7 +205,7 @@ optional arguments:
### Basic usage
```bash
# Run inference with the quantized model
python run_inference.py -m models/Falcon3-7B-Instruct-1.58bit/ggml-model-i2_s.gguf -p "You are a helpful assistant" -cnv
python run_inference.py -m models/BitNet-b1.58-2B-4T/ggml-model-i2_s.gguf -p "You are a helpful assistant" -cnv
```
<pre>
usage: run_inference.py [-h] [-m MODEL] [-n N_PREDICT] -p PROMPT [-t THREADS] [-c CTX_SIZE] [-temp TEMPERATURE] [-cnv]
Expand Down Expand Up @@ -329,5 +277,36 @@ python utils/generate-dummy-bitnet-model.py models/bitnet_b1_58-large --outfile
# Run benchmark with the generated model, use -m to specify the model path, -p to specify the prompt processed, -n to specify the number of token to generate
python utils/e2e_benchmark.py -m models/dummy-bitnet-125m.tl1.gguf -p 512 -n 128
```
### FAQ (Frequently Asked Questions)📌

#### Q1: The build dies with errors building llama.cpp due to issues with std::chrono in log.cpp?

**A:**
This is an issue introduced in recent version of llama.cpp. Please refer to this [commit](https://github.com/tinglou/llama.cpp/commit/4e3db1e3d78cc1bcd22bcb3af54bd2a4628dd323) in the [discussion](https://github.com/abetlen/llama-cpp-python/issues/1942) to fix this issue.

#### Q2: How to build with clang in conda environment on windows?

**A:**
Before building the project, verify your clang installation and access to Visual Studio tools by running:
```
clang -v
```

This command checks that you are using the correct version of clang and that the Visual Studio tools are available. If you see an error message such as:
```
'clang' is not recognized as an internal or external command, operable program or batch file.
```

It indicates that your command line window is not properly initialized for Visual Studio tools.

• If you are using Command Prompt, run:
```
"C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\Tools\VsDevCmd.bat" -startdir=none -arch=x64 -host_arch=x64
```

• If you are using Windows PowerShell, run the following commands:
```
Import-Module "C:\Program Files\Microsoft Visual Studio\2022\Professional\Common7\Tools\Microsoft.VisualStudio.DevShell.dll" Enter-VsDevShell 3f0e31ad -SkipAutomaticLocation -DevCmdArguments "-arch=x64 -host_arch=x64"
```

These steps will initialize your environment and allow you to use the correct Visual Studio tools.
Binary file added assets/header_model_release.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
9 changes: 8 additions & 1 deletion setup_env.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,9 @@
"tiiuae/Falcon3-1B-Instruct-1.58bit": {
"model_name": "Falcon3-1B-Instruct-1.58bit",
},
"microsoft/BitNet-b1.58-2B-4T": {
"model_name": "BitNet-b1.58-2B-4T",
},
}

SUPPORTED_QUANT_TYPES = {
Expand Down Expand Up @@ -161,6 +164,8 @@ def gen_code():
run_command([sys.executable, "utils/codegen_tl1.py", "--model", "Llama3-8B-1.58-100B-tokens", "--BM", "256,128,256,128", "--BK", "128,64,128,64", "--bm", "32,64,32,64"], log_step="codegen")
elif get_model_name() == "bitnet_b1_58-3B":
run_command([sys.executable, "utils/codegen_tl1.py", "--model", "bitnet_b1_58-3B", "--BM", "160,320,320", "--BK", "64,128,64", "--bm", "32,64,32"], log_step="codegen")
elif get_model_name() == "BitNet-b1.58-2B-4T":
run_command([sys.executable, "utils/codegen_tl1.py", "--model", "bitnet_b1_58-3B", "--BM", "160,320,320", "--BK", "64,128,64", "--bm", "32,64,32"], log_step="codegen")
else:
raise NotImplementedError()
else:
Expand All @@ -177,6 +182,8 @@ def gen_code():
run_command([sys.executable, "utils/codegen_tl2.py", "--model", "Llama3-8B-1.58-100B-tokens", "--BM", "256,128,256,128", "--BK", "96,96,96,96", "--bm", "32,32,32,32"], log_step="codegen")
elif get_model_name() == "bitnet_b1_58-3B":
run_command([sys.executable, "utils/codegen_tl2.py", "--model", "bitnet_b1_58-3B", "--BM", "160,320,320", "--BK", "96,96,96", "--bm", "32,32,32"], log_step="codegen")
elif get_model_name() == "BitNet-b1.58-2B-4T":
run_command([sys.executable, "utils/codegen_tl2.py", "--model", "bitnet_b1_58-3B", "--BM", "160,320,320", "--BK", "96,96,96", "--bm", "32,32,32"], log_step="codegen")
else:
raise NotImplementedError()

Expand Down Expand Up @@ -222,4 +229,4 @@ def signal_handler(sig, frame):
args = parse_args()
Path(args.log_dir).mkdir(parents=True, exist_ok=True)
logging.basicConfig(level=logging.INFO)
main()
main()
Loading