Skip to content

Run CI unittests in parallel #3445

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Mar 1, 2021
Merged

Run CI unittests in parallel #3445

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
51df4aa
enable parallel tests
pmeier Feb 24, 2021
15d60c6
disable parallelism for GPU tests
pmeier Feb 24, 2021
b0670f1
[test] limit maximum processes on linux
pmeier Feb 24, 2021
bc05f7c
[debug] limit max processes even further
pmeier Feb 24, 2021
754f3c0
[test] use subprocesses over threads
pmeier Feb 24, 2021
6c01b2d
[test] limit intra-op threads
pmeier Feb 24, 2021
654b23f
only limit intra op threads for CPU tests
pmeier Feb 24, 2021
641051f
[poc] use low timeout for showcasing
pmeier Feb 25, 2021
6c6b478
[poc] fix syntax
pmeier Feb 25, 2021
1bab2a6
Merge branch 'master' into pytest-xdist
fmassa Mar 1, 2021
677c447
set timeout to 5 minutes
pmeier Mar 1, 2021
7eac5e5
Merge branch 'master' into pytest-xdist
pmeier Mar 1, 2021
e2803e8
Merge remote-tracking branch 'pmeier/pytest-xdist' into pytest-xdist
pmeier Mar 1, 2021
70926d2
fix timeout on windows
pmeier Mar 1, 2021
25b118d
Merge branch 'master' into pytest-xdist
fmassa Mar 1, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .circleci/unittest/linux/scripts/environment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ channels:
- conda-forge
dependencies:
- pytest
- pytest-xdist
- pytest-timeout
- pytest-cov
- codecov
- pip
Expand Down
17 changes: 16 additions & 1 deletion .circleci/unittest/linux/scripts/run_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,20 @@ eval "$(./conda/bin/conda shell.bash hook)"
conda activate ./env

export PYTORCH_TEST_WITH_SLOW='1'
if [ "${CU_VERSION:-}" == cpu ] ; then
NUMPROCESSES="auto"
export OMP_NUM_THREADS="1"
else
NUMPROCESSES="1"
fi

python -m torch.utils.collect_env
pytest --cov=torchvision --junitxml=test-results/junit.xml -v --durations 20 test --ignore=test/test_datasets_download.py
pytest \
--numprocesses=$NUMPROCESSES \
--timeout=300 \
--cov=torchvision \
--junitxml=test-results/junit.xml \
--verbose \
--durations 20 \
--ignore=test/test_datasets_download.py \
test
2 changes: 2 additions & 0 deletions .circleci/unittest/windows/scripts/environment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ channels:
- conda-forge
dependencies:
- pytest
- pytest-xdist
- pytest-timeout
- pytest-cov
- codecov
- pip
Expand Down
17 changes: 16 additions & 1 deletion .circleci/unittest/windows/scripts/run_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,20 @@ eval "$(./conda/Scripts/conda.exe 'shell.bash' 'hook')"
conda activate ./env

export PYTORCH_TEST_WITH_SLOW='1'
if [ "${CU_VERSION:-}" == cpu ] ; then
NUMPROCESSES="auto"
export OMP_NUM_THREADS="1"
else
NUMPROCESSES="1"
fi

python -m torch.utils.collect_env
pytest --cov=torchvision --junitxml=test-results/junit.xml -v --durations 20 test --ignore=test/test_datasets_download.py
pytest \
--numprocesses=$NUMPROCESSES \
--timeout=300 \
--cov=torchvision \
--junitxml=test-results/junit.xml \
--verbose \
--durations 20 \
--ignore=test/test_datasets_download.py \
test