update deepspeed docker #37371
update deepspeed docker #37371
Conversation
|
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| name: "Latest PyTorch + DeepSpeed" | ||
| runs-on: | ||
| group: aws-general-8-plus | ||
| group: aws-g4dn-2xlarge-cache |
There was a problem hiding this comment.
this is the real change in this file. other places are just space issues.
There was a problem hiding this comment.
we need this otherwise the build fails. No idea, looks like g++ versions issues.
|
|
||
| RUN python3 -m pip install --no-cache-dir ./transformers[deepspeed-testing] | ||
| # `datasets` requires pandas, pandas has some modules compiled with numpy=1.x causing errors | ||
| RUN python3 -m pip install --no-cache-dir './transformers[deepspeed-testing]' 'pandas<2' 'numpy<2' |
There was a problem hiding this comment.
nightly image is not the current focus, but I went forward and the build works locally.
Let's update the docker file anyway.
There was a problem hiding this comment.
yeah but i think that we are not really installing deepspeed from source in this dockerfile. But as you said, this is not the current focus and not sure this is needed for now
|
Thanks for your help @ydshieh ! |
* update * create docker image * 03 * uninstall pytest as it conflits with transformers * wrong one * better * see which package depends on pytest * up * resintall * fix * deepspeedddddddd * deepspeedddddddd * deepspeedddddddd * deepspeedddddddd * deepspeedddddddd * deepspeedddddddd * deepspeedddddddd * deepspeedddddddd --------- Co-authored-by: ydshieh <[email protected]>
* update * create docker image * 03 * uninstall pytest as it conflits with transformers * wrong one * better * see which package depends on pytest * up * resintall * fix * deepspeedddddddd * deepspeedddddddd * deepspeedddddddd * deepspeedddddddd * deepspeedddddddd * deepspeedddddddd * deepspeedddddddd * deepspeedddddddd --------- Co-authored-by: ydshieh <[email protected]>
What does this PR do?
This PR updates deepspeed docker so that we run our tests with pytorch 2.6