Test torchprime from PyTorch/XLA #9152
Conversation
tengyifei
force-pushed
the
yifeit/torchprime-ci-3
branch
6 times, most recently
from
3dad488 to
9be6f35
Compare
tengyifei
force-pushed
the
yifeit/torchprime-ci-3
branch
from
8c05962 to
96c6541
Compare
tengyifei
requested review from
lsy323,
ManfeiBai,
zpcore,
bhavya01 and
qihqi
as code owners
| @@ -0,0 +1,23 @@ | |||
| # syntax=docker/dockerfile:1.4 | |||
There was a problem hiding this comment.
Document the purpose of this dockerfile (e.g. make it clear this is just for torchprime testing, not used by torchprime itself in its normal usage)?
There was a problem hiding this comment.
Consider renaming this to torchprime_e2e_test.Dockerfile to make the purpose clear?
| FROM python:${python_version}-${debian_version} AS release | ||
|
|
||
| WORKDIR /tmp/wheels | ||
| COPY ./*.whl ./ |
There was a problem hiding this comment.
Document what these wheels are?
| @@ -0,0 +1,63 @@ | |||
| #!/bin/bash | |||
There was a problem hiding this comment.
Consider renaming to publish_torchprime_e2e_test_docker.sh for clarity?
| sudo usermod -aG docker $USER | ||
| newgrp docker | ||
| shell: bash | ||
| # Googlers: if this fails, follow http://shortn/_61iSj31q1b to debug. |
There was a problem hiding this comment.
Nit: use a go/ link?
| DOCKER_IMAGE_TAG: ${{ steps.random_tag.outputs.uuid }} | ||
| DOCKER_PROJECT: tpu-pytorch | ||
| # Trigger torchprime E2E test workflow | ||
| - uses: convictional/[email protected] |
There was a problem hiding this comment.
Can we document how to debug this workflow and get help if needed?
| DOCKER_IMAGE_TAG: ${{ steps.random_tag.outputs.uuid }} | ||
| DOCKER_PROJECT: tpu-pytorch | ||
| # Trigger torchprime E2E test workflow | ||
| - uses: convictional/[email protected] |
There was a problem hiding this comment.
Who provisions for this workflow? Does it have enough capacity for our needs?
This PR fixes AI-Hypercomputer/torchprime#161. See issue description for motivation and details. While don't quite have HBM usage check and loss curve checks yet, those could be added later.
This PR adds a new workflow that will be run during pre- and post-submits:
This will let us see if PyTorch/XLA PRs break model training in torchprime. I intend to first run this check as FYI-only for a while, and then later we could promote it to "required" to gate landing of PRs if it's very stable.