Use common base class private functions for TikToken#45

Merged

larryliu0820 merged 1 commit intopytorch-labs:mainfrom

jackzhxng:jz/reuse

Apr 15, 2025

Commits on Apr 14, 2025

TikToken uses common bpe base functions
jackzhxng
committed