Skip to content

Improve experience with TOKENIZERS_PARALLELISM #328

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
n1t0 opened this issue Jul 6, 2020 · 2 comments · Fixed by #329
Closed

Improve experience with TOKENIZERS_PARALLELISM #328

n1t0 opened this issue Jul 6, 2020 · 2 comments · Fixed by #329

Comments

@n1t0
Copy link
Contributor

n1t0 commented Jul 6, 2020

The message is not explicit enough, so some users don't understand that they should set an environment variable. Also, we could probably provide a function that, given a boolean, would set it for the user.

@cornzz
Copy link

cornzz commented Oct 1, 2024

Hi, I am using the zeroscrolls metric to evaluate something and I am getting this message, I have no idea if I should set this variable to true or false since I didnt write the code using the tokenizer and also dont know where this is coming from because the message doesnt tell that. What do I do?

@ArthurZucker
Copy link
Collaborator

You can probably ignore it, or do something like export TOKENIZER_PARALLELISM=1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants