Skip to content

Add experimental BF16 support for Metal #809

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
alvarobartt wants to merge 28 commits into
base: main
Choose a base branch
from
Draft

Add experimental BF16 support for Metal #809

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
4009d62
Add support for BF16 w/ `candle` (WIP)
alvarobartt Jan 20, 2026
4a9400e
Add support for `voyageai/voyage-4-*` (WIP)
alvarobartt Jan 20, 2026
e9b9332
Remove `half` crate dependency
alvarobartt Jan 20, 2026
d4f66bf
Add `tracing::warn!` on `DType::Bfloat16`
alvarobartt Jan 21, 2026
db7df49
Update `channel` to 1.92 in `rust-toolchain.toml`
alvarobartt Jan 21, 2026
6b2a1f1
Constrain `Bfloat16` on `candle` for Metal and CUDA
alvarobartt Jan 31, 2026
cec5de8
Update `tracing::warn!` for `DType::Bfloat16`
alvarobartt Jan 31, 2026
8889979
Fix `DType` enum and impl (never default to BF16)
alvarobartt Jan 31, 2026
35dc4d7
Add minimum finite value for `DType::BF16`
alvarobartt Jan 31, 2026
d7962ed
Restore `qwen3.rs` prior Voyage AI related changes
alvarobartt Jan 31, 2026
92c6ae7
Use `actions-rust-lang` and fix typo in "linting"
alvarobartt Jan 31, 2026
3838c04
Fix formatting in YAML files
alvarobartt Jan 31, 2026
04f5d63
Set default minimum finite value to FP16 value
alvarobartt Jan 31, 2026
b1a1b23
Calculate distance in Gemma3 w/ `abs_diff` instead (clippy)
alvarobartt Jan 31, 2026
b547b0b
Fix feature-gating for `DType::Bfloat16` (and exclude Turing and Volta)
alvarobartt Jan 31, 2026
43a8e99
Add note on lack of BF16 support for Turing (and earlier)
alvarobartt Jan 31, 2026
66b6a7b
Use `into_iter` for `Sequence` as `get_pre_tokenizers` unavailable
alvarobartt Jan 31, 2026
07fe2e7
Add runtime validation on CUDA compute cap for BF16
alvarobartt Jan 31, 2026
e506c93
Update `rustc` to latest stable 1.92
alvarobartt Jan 31, 2026
e3e34b8
Merge branch 'main' into add-bfloat16-support
alvarobartt Jan 31, 2026
eb4f13b
Revert `Default` impl for `DType` when `feature = "python"`
alvarobartt Jan 31, 2026
9745b0e
Add missing `cuda` feature flag to `candle-cuda` and `candle-cuda-tur…
alvarobartt Feb 1, 2026
9387804
Add BF16 support for `FlashQwen3`
alvarobartt Feb 1, 2026
29fe799
Remove `feature = "python"` from `tracing:warn!` on BF16
alvarobartt Feb 1, 2026
df60701
Add `supports_flash_attn` to remove duplicated code
alvarobartt Feb 1, 2026
b83102a
Update `index_select` to exclude CUDA + BF16
alvarobartt Feb 2, 2026
f6880c7
Skip BF16 support for CUDA (only Metal)
alvarobartt Feb 2, 2026
c19087e
Fix `index_select` feature gating
alvarobartt Feb 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .github/workflows/build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ on:
workflow_dispatch:
push:
branches:
- 'main'
- "main"
tags:
- 'v*'
- "v*"
pull_request:
paths:
- ".github/workflows/build.yaml"
Expand All @@ -19,7 +19,7 @@ on:
- "rust-toolchain.toml"
- "Dockerfile"
branches:
- 'main'
- "main"

jobs:
matrix:
Expand Down Expand Up @@ -126,7 +126,7 @@ jobs:
context: .
file: ${{ matrix.dockerfile }}
push: ${{ github.event_name != 'pull_request' }}
platforms: 'linux/amd64'
platforms: "linux/amd64"
build-args: |
SCCACHE_GHA_ENABLED=${{ matrix.sccache }}
CUDA_COMPUTE_CAP=${{ matrix.cudaComputeCap }}
Expand Down Expand Up @@ -168,7 +168,7 @@ jobs:
target: grpc
file: ${{ matrix.dockerfile }}
push: ${{ github.event_name != 'pull_request' }}
platforms: 'linux/amd64'
platforms: "linux/amd64"
build-args: |
SCCACHE_GHA_ENABLED=${{ matrix.sccache }}
CUDA_COMPUTE_CAP=${{ matrix.cudaComputeCap }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/integration-test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ name: Run integration tests
on:
workflow_dispatch:
schedule:
- cron: '0 0 * * *' # Run the workflow nightly to check Gaudi integration is working
- cron: "0 0 * * *" # Run the workflow nightly to check Gaudi integration is working

jobs:
tests:
Expand Down
7 changes: 1 addition & 6 deletions .github/workflows/liniting.yaml → .github/workflows/linting.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,8 @@ jobs:
steps:
- uses: actions/checkout@v6
- name: Install Rust
uses: actions-rs/toolchain@v1
uses: actions-rust-lang/setup-rust-toolchain@v1.15.2
with:
# Released on: 28 December, 2023
# Branched from master on: 10 November, 2023
# https://releases.rs/docs/1.85.0/
toolchain: 1.85.0
override: true
components: rustfmt, clippy
- name: Install Protoc
uses: arduino/setup-protoc@v1
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v6
- uses: actions-rust-lang/setup-rust-toolchain@v1
- uses: actions-rust-lang/setup-rust-toolchain@v1.15.2
- name: Run sccache-cache
uses: mozilla-actions/sccache-action@v0.0.9
with:
Expand Down
Loading
Loading