Skip to content

Conversation

@gante
Copy link
Contributor

@gante gante commented Jul 8, 2025

What does this PR do?

Adds a tokenizer mapping for smollm3, which fixes the following script

from transformers import pipeline

# tokenizer is infered from the model, through the tokenizer mapping
pipe = pipeline("text-generation", model="HuggingFaceTB/SmolLM3-3B-Base")
pipe("Once upon a time,")

@gante gante requested review from ArthurZucker and anton-l July 8, 2025 09:54
@gante gante enabled auto-merge (squash) July 8, 2025 10:28
@github-actions
Copy link
Contributor

github-actions bot commented Jul 8, 2025

[For maintainers] Suggested jobs to run (before merge)

run-slow: auto

@gante gante merged commit 1580f64 into huggingface:main Jul 8, 2025
25 checks passed
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@gante gante deleted the smollm3_tok_mapping branch July 8, 2025 10:48
@Vaibhavs10 Vaibhavs10 added the for patch Tag issues / labels that should be included in the next patch label Jul 8, 2025
Cyrilvallez pushed a commit that referenced this pull request Jul 11, 2025
rjgleaton pushed a commit to rjgleaton/transformers that referenced this pull request Jul 17, 2025
zaristei pushed a commit to zaristei/transformers that referenced this pull request Sep 9, 2025
zaristei pushed a commit to zaristei/transformers that referenced this pull request Sep 9, 2025
zaristei pushed a commit to zaristei/transformers that referenced this pull request Sep 9, 2025
zaristei pushed a commit to zaristei/transformers that referenced this pull request Sep 9, 2025
zaristei pushed a commit to zaristei/transformers that referenced this pull request Sep 9, 2025
zaristei pushed a commit to zaristei/transformers that referenced this pull request Sep 9, 2025
zaristei pushed a commit to zaristei/transformers that referenced this pull request Sep 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

for patch Tag issues / labels that should be included in the next patch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants