Hanging in TextClassificationPipeline's prediction

[amiralikaboli]

System Info

• transformers version: 4.22.2
• Platform: Linux-5.15.0-52-generic-x86_64-with-glibc2.35
• Python version: 3.8.15
• Huggingface_hub version: 0.10.0
• PyTorch version (GPU?): 1.7.1 (False)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

Who can help?

@Narsil @sgugger

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

I am using this class. It hangs forever after deploying and sending a request to my server (using falcon and gunicorn) that calls the predict function. However, when I call it in a simple script, everything is ok, and its predictions are returned.

class TrainStage:
    def __init__(self, config):
    	self.config = config

    def fit(self):
        model_config = AutoConfig.from_pretrained(self.config.pretrained_model_path)
        self.tokenizer = AutoTokenizer.from_pretrained(self.config.pretrained_model_path)
        self.model = AutoModelForSequenceClassification.from_pretrained(
            self.config.pretrained_model_path, 
            config=model_config
        )

        training_args = TrainingArguments(...)
        trainer = Trainer(...)
        trainer.train()

    def transform(self, texts: List[str]):
        pipeline = TextClassificationPipeline(model=self.model, tokenizer=self.tokenizer)
        results = pipeline(texts)
        return results

Expected behavior

Returning predictions

[Narsil]

@amiralikaboli Can you try setting TOKENIZERS_PARALLELISM=0 before calling your script ?

You might be triggerring: #5486

Basically tokenizers does parallelism by default, but it can be messed up by other sources of parallelism, most cases are handled, but maybe you found a way to trigger the deadlock.
Using that might help at least make sure this is not the issue.

And if the problem is still there, could you provide a simple reproducing script ?

[amiralikaboli]

@Narsil Actually, It doesn't work. The script I use for training and predicting my model is similar to the above script. Do you want a script for a server/client which runs the model?

[Narsil]

If you could provide a simple (one file) script that's easily launchable to reproduce that'd be perfect yes.

The bug you're encountering is almost certainly a deadlock linked to multiple libs doing parallelism in some way hurting each other. Without the full script to reproduce it's hard to pinpoint though. Also are you on Mac or Linux (there's different default behavior for forking if my memory serves correctly).

[amiralikaboli]

You are right about the deadlock. We loaded our model several times as a temporary solution, and it worked. But, it is not ideal because of more resource usage.
Providing you with a simple script exactly based on our case is not possible since we used a private library over falcon.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.