Feature request
The current behavior has hardcoded options for the tokenizer https://github.com/huggingface/transformers.js/blob/main/src/pipelines.js#L761-L775
Motivation
It would be nice to stick with the generic pipeline API and be able to pass through custom tokenizer options that are overwriting tokenizer.json
Your contribution
If the idea sounds good I can contribute the patch
Feature request
The current behavior has hardcoded options for the tokenizer https://github.com/huggingface/transformers.js/blob/main/src/pipelines.js#L761-L775
Motivation
It would be nice to stick with the generic
pipelineAPI and be able to pass through custom tokenizer options that are overwritingtokenizer.jsonYour contribution
If the idea sounds good I can contribute the patch