Skip to content

Allow configuring the tokenizer in Text2TextGenerationPipeline #1096

Description

@tarekziade

Feature request

The current behavior has hardcoded options for the tokenizer https://github.com/huggingface/transformers.js/blob/main/src/pipelines.js#L761-L775

Motivation

It would be nice to stick with the generic pipeline API and be able to pass through custom tokenizer options that are overwriting tokenizer.json

Your contribution

If the idea sounds good I can contribute the patch

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions