Tokenizers throwing warning "The current process just got forked, Disabling parallelism to avoid deadlocks.. To disable this warning, please explicitly set TOKENIZERS_PARALLELISM=(true | false)"

[saahiluppal]

I know this warning is because the transformer library is updated to 3.x.
I know the warning saying to set TOKENIZERS_PARALLELISM = true / false

My question is where should i set TOKENIZERS_PARALLELISM = true / false
is this when defining tokenizers like

tok = Tokenizer.from_pretrained('xyz', TOKENIZERS_PARALLELISM=True) // this doesn't work

or is this when encoding text like

tok.encode_plus(text_string, some=some, some=some, TOKENIZERS_PARALLELISM = True) // this also didn't work

Suggestions anyone?

[hadarishav]

This might help you: https://stackoverflow.com/questions/62691279/how-to-disable-tokenizers-parallelism-true-false-warning

[Vimos]

I suspect this may be caused by loading data. In my case, it happens when my dataloader starts working.

[n1t0]

This is happening whenever you use multiprocessing (Often used by data loaders). The way to disable this warning is to set the TOKENIZERS_PARALLELISM environment variable to the value that makes more sense for you. By default, we disable the parallelism to avoid any hidden deadlock that would be hard to debug, but you might be totally fine while keeping it enabled in your specific use-case.

You can try to set it to true, and if your process seems to be stuck, doing nothing, then you should use false.

We'll improve this message to help avoid any confusion (Cf huggingface/tokenizers#328)

[nathan-chappell]

I may be a rookie, but it seems like it would be useful to indicate that this is an environment variable in the warning message.

[n1t0]

You are totally right! In the latest version 3.0.2, the warning message should be a lot better, and it will trigger only when necessary.

[ierezell]

Hi, sorry to bump this thread...

I'm having the same problem however, the tokenizer is used only in my model.

Data loading is made with multiple workers but it is only loading raw text which is then given to the model and only the model uses the tokenizer.
I don't have multi model or whatever, just a classic pytorch model.

Thus I was wondering how can I have the warning.

Thanks in advance,
Have a great day :)

[n1t0]

You must be using a tokenizer before using multiprocessing. When your process gets forked, you see this message because it detects that a fork is happening and that some kind of parallelism was used before.

[ierezell]

@n1t0,
Thanks a lot for the fast reply,
I guess it detect a fork even if it's safe for me to do so... Yes my process is forked but not the tokenizer.

Then I will use the env variable to remove the warning.

[ritwikmishra]

I use tokenizer in my data loader.

If that is the source of this problem (hence disabling the parallelization --> hence slow training), then what is the solution?

Using tokenizer in the pre-processing step?

[hbchen121]

After testing, it is found that when the data in a dataloader is processed by the token, and the datalodaer jumps out before it is finished, this warning will be triggered;
I give a code example:

# for example, following code will trigger the warning
for texts in train_dataloader:
    _ = tokenizer.batch_encode_plus(texts)
    # loader has not been traversed
    # but texts are used
    break 
for texts in test_dataloader:
    # warning ...
    pass or break

# and following code will not trigger the warning
for texts in train_dataloader:
    # loader has not been traversed
    # but texts are not used
    break 
for texts in test_dataloader:
    # No warning 
    pass or break

[ritwikmishra]

@hbchen121 my dataloader processes the text in init function

During data loading time, directly input_ids and attention masks are fetched, yet I get this warning.

[Jadiker]

Despite the documentation saying that use_fast defaults to False, adding use_fast=False so that it's AutoTokenizer.from_pretrained(model_name, use_fast=False) removed this warning for me. If I just use AutoTokenizer.from_pretrained(model_name), the warning pops up again.

[hzphzp]

I want to know if we can ignore this warning. What bad effects will it have? Will it affect the training results? Or is it just a little slower? If the environment variables are changed according to the above solution, what is the cost of doing so?

[amyeroberts]

cc @ArthurZucker

[crmuhsin]

I want to know if we can ignore this warning. What bad effects will it have? Will it affect the training results? Or is it just a little slower? If the environment variables are changed according to the above solution, what is the cost of doing so?

@hzphzp there is an explanation in SO
https://stackoverflow.com/questions/62691279/how-to-disable-tokenizers-parallelism-true-false-warning/72926996#72926996

[hzphzp]

I want to know if we can ignore this warning. What bad effects will it have? Will it affect the training results? Or is it just a little slower? If the environment variables are changed according to the above solution, what is the cost of doing so?

@hzphzp there is an explanation in SO https://stackoverflow.com/questions/62691279/how-to-disable-tokenizers-parallelism-true-false-warning/72926996#72926996

Thank you!

[ctwardy]

Though each notebook runs fine by itself, I get this warning when running multiple notebooks via nbdev_test (https://github.com/fastai/nbdev). Shortly afterwards it crashes due to out-of-memory.

I assume it has something to do with multiprocessing in nbdev_test, even when setting --n_workers 1.

This gets a warning about disabling parallelism to avoid locks:

nbdev_test --n_workers 1 --pause 10 --do_print --file_glob "*nb"

This works fine:

$ for x in `ls nbs/*nb`; do nbdev_test --n_workers 1 --do_print --path "$x"; done

[HoseinHashemi]

Despite the documentation saying that use_fast defaults to False, adding use_fast=False so that it's AutoTokenizer.from_pretrained(model_name, use_fast=False) removed this warning for me. If I just use AutoTokenizer.from_pretrained(model_name), the warning pops up again.

use_fast default is True, which enables a fast Rust base tokeniser if available, if not a Python base tokeniser will be triggered.