Skip to content

Switch to BPE tokenization impl from openai/tiktoken#1118

Open
iceychris wants to merge 1 commit intoggml-org:masterfrom
iceychris:fix-tokenization
Open

Switch to BPE tokenization impl from openai/tiktoken#1118
iceychris wants to merge 1 commit intoggml-org:masterfrom
iceychris:fix-tokenization

Commits

Commits on Jul 18, 2023