enable trainer test cases on xpu

[yao-matrix]

No description provided.

[github-actions]

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the Ready for review button (at the bottom of the PR page). This will assign reviewers and trigger CI.

[yao-matrix]

@ydshieh @IlyasMoutawwakil , pls help review, thx.

[yao-matrix]

There is an implicit issue here.

1. In the env where PYTHONPATH is empty, the line 2049 code join src and test to PYTHONPATH with an extra ":"(since paths has a "" string because of the code paths.append(env.get("PYTHONPATH", ""))), so the final PYTHONPATH will be like /transformers/src:/transformers/tests: with a trailing :, which implies the current folder which is /transformers.
2. But when it comes the an env with an non-empty PYTHONPATH, the env["PYTHONPATH"] = ":".join(paths) will generate /transformers/src:/transformers/tests:A:B path without training :, which means current folder not included in the PYTHONPATH

So, in situation 2), test cases like tests/trainer/test_trainer_distributed_loss.py will fail, since while torchrun, the from tests.trainer.test_trainer import StoreLossCallback will not found, since there is only /transformers/src:/transformers/tests in PYTHONPATH, but the import statement actually needs /transformers/

what i do is making it explicit, but this is just for the cases i can see, i don't know other background.

[ydshieh]

SGTM. But just to understand better,

when it comes the an env with an non-empty PYTHONPATH

So in your environment, this is the case? Do you set it somehow?

In our report, I see

tests/trainer/test_trainer_distributed_loss.py::TestTrainerDistributedLoss::test_trainer
(line 675) AssertionError: 0.8057947486768737 not less than 0.1

So it seems there is no import issue in our runner env.

[yao-matrix]

@ydshieh yes, in my CUDA env, there is no such issue either, since PYTHONPATH is empty. But in my XPU env, since Intel oneAPI will preset some paths into PYTHONPATH, and it doesn't work. That's the background i looked into that issue and found it,:)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.