Add option to disable EOS appending in preprocessing (#5615)#5616

surprisedPikachu007 wants to merge 1 commit intofacebookresearch:mainfacebookresearch/fairseq:mainfrom

surprisedPikachu007:patch-preprocessingsurprisedPikachu007/fairseq:patch-preprocessingCopy head branch name to clipboard