This repository was archived by the owner on Mar 20, 2026. It is now read-only.
Repository navigation
This repository was archived by the owner on Mar 20, 2026. It is now read-only.
Add an option to disable appending eos to the end of each document while preprocessing #5615
Description
🚀 Feature Request
Add an option to disable appending EOS to the end of each document while preprocessing
Motivation
There are instances where adding EOS is not preferred like when packing data to sequence length.
Pitch
Proposing a change to add an argument to handle this
Activity
Add option to disable EOS appending in preprocessing (facebookresearc…