Use default timeout of 30 minutes for gloo backend #13056
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
10s seems too little, but 30m seems a bit too much, no? something like 1 min? |
|
@soumith There are instances where a single process group is used for a barrier between 2+ hour asynchronous tasks that complete with some variance (e.g. +/- 10m). I'm working on supporting per operation timeouts that will allow you to explicitly set a timeout on a barrier to do this, but until then we have to use a single timeout for all ops. |
|
@pietern interesting! |
pietern
force-pushed
the
export-D10558746
branch
from
1d92a7f to
2f4469c
Compare
pietern
force-pushed
the
export-D10558746
branch
from
2f4469c to
96abcad
Compare
pietern
force-pushed
the
export-D10558746
branch
from
96abcad to
1238c38
Compare
pietern
force-pushed
the
export-D10558746
branch
from
1238c38 to
dfe33c2
Compare
There was a problem hiding this comment.
pietern is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
pietern
force-pushed
the
export-D10558746
branch
from
dfe33c2 to
16067c2
Compare
pietern
force-pushed
the
export-D10558746
branch
from
16067c2 to
93e35c1
Compare
Summary: The existing default timeout was set at 10 seconds, which is too low for asynchronous tasks that depend on a barrier to resynchronize. Having a single timeout for all operations is not ideal and this will be addressed in future commits. Differential Revision: D10558746 fbshipit-source-id: 26c811da27907c1388c40d6f3a352ba96a011c15
pietern
force-pushed
the
export-D10558746
branch
from
93e35c1 to
c676c46
Compare
There was a problem hiding this comment.
pietern is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
The existing default timeout was set at 10 seconds, which is too low
for asynchronous tasks that depend on a barrier to resynchronize.
Having a single timeout for all operations is not ideal and this will
be addressed in future commits.
Differential Revision: D10558746