Skip to content

process group timeout from 600s to 1200s - #4081

Closed
benenzhu wants to merge 1 commit into
ROCm:mainfrom
RadeonFlow:longer_timeout
Closed

benenzhu wants to merge 1 commit into
ROCm:mainfrom
RadeonFlow:longer_timeout

Conversation

@benenzhu

@benenzhu benenzhu commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Motivation

This can reslove an issue that ATOM's Kimi model load may randomly failed due to the slow ssds, and the broadcast ended timeouts.

https://github.com/SemiAnalysisAI/InferenceX/actions/runs/28670767395/job/85033302286#step:4:419

Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/app/ATOM/atom/model_engine/async_proc.py", line 130, in __init__
    self.runners = [runner_class(rank, *args, **kwargs)]
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/ATOM/atom/model_engine/model_runner.py", line 736, in __init__
    self.warmup_model()
  File "/app/ATOM/atom/model_engine/model_runner.py", line 1169, in warmup_model
    self.forward(dummy_batch)
  File "/opt/venv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/app/ATOM/atom/model_engine/model_runner.py", line 2193, in forward
    fwd_output = self.postprocess(
                 ^^^^^^^^^^^^^^^^^
  File "/app/ATOM/atom/model_engine/model_runner.py", line 2112, in postprocess
    sampled_tokens = get_tp_group().broadcast(sampled_tokens, src=0)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/aiter-test/aiter/dist/parallel_state.py", line 1018, in broadcast
    torch.distributed.broadcast(
  File "/opt/venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 2907, in broadcast
    work = group.broadcast([tensor], opts)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.distributed.DistBackendError: [2] is setting up NCCL communicator and retrieving ncclUniqueId from [0] via c10d key-value store by key '0', but store->get('0') got error: wait timeout after 600000ms, keys: /default_pg/0//3//cuda//0

Technical Details

Change the timeout from 600s to 1200s.
Should not effect any correctness.

Test Plan

Test Result

Submission Checklist

@benenzhu
benenzhu requested a review from a team July 5, 2026 03:58
@benenzhu benenzhu changed the title process group timeout from 600 to 1200 process group timeout from 600s to 1200s Jul 5, 2026
@github-actions

github-actions Bot commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

🏷️ CI Guide

Runs automatically on every PR:

  • ✅ Pre-checks (submodule verification, code formatting)
  • ✅ Aiter op tests (gfx942 + gfx950)
  • ✅ Triton tests on MI35X (only when aiter/ops/triton/** or related paths are changed)

Extended tests (opt-in via labels):

Label Tests
ci:triton-300x Run an additional Triton test job on MI300X in PRs; main branch always runs both MI35X and MI300X
ci:sglang SGLang integration tests: DeepSeek-R1-MXFP4 accuracy, Qwen 3.5 accuracy
ci:atom ATOM benchmark: DeepSeek-R1-0528, GPT-OSS-120B
ci:atom_full ATOM accuracy suite for PR and main models from ATOM models_accuracy.json
ci:vllm vLLM benchmark: GPT-OSS-120B, DeepSeek-R1-0528, Kimi-K2.5
ci:all All standard extended tests (excludes ci:atom_full)

Only add ci:atom_full for FlyDSL or Triton upgrades.
Add labels via the sidebar or gh pr edit 4081 --add-label <label>

@benenzhu

Copy link
Copy Markdown
Contributor Author

Closed cause it can be sloved by ROCm/ATOM#1465 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant