Some tokens might legitimately be the single 0xFF byte. This can be determined by checking if the length of the token is equal to 1.
I have found that tokenizing "\xFF" gives the correct token id. While decoding that token id in a round trip test simply gives back an empty string.
Fixing the decoding step seems rather trivial by changing this line to check that tok.len() > 1 like so
else if t[0] == TokTrie::SPECIAL_TOKEN_MARKER && t.len() > 1
This makes my round trip tests pass.
Note: Ordering the special token check before the length check to short circuit the branch statements more consistently. The length check is already covered by the is_empty() case a few lines earlier.
Some tokens might legitimately be the single 0xFF byte. This can be determined by checking if the length of the token is equal to 1.
I have found that tokenizing "\xFF" gives the correct token id. While decoding that token id in a round trip test simply gives back an empty string.
Fixing the decoding step seems rather trivial by changing this line to check that
tok.len() > 1like soThis makes my round trip tests pass.
Note: Ordering the special token check before the length check to short circuit the branch statements more consistently. The length check is already covered by the is_empty() case a few lines earlier.