Skip to content

Fix Deepseek Qwen Fatal Bus Error [#291] - #456

Merged
ddilbazTT merged 2 commits into
mainfrom
deepseek_qwen
Mar 19, 2025
Merged

ddilbazTT merged 2 commits into
mainfrom
deepseek_qwen

Conversation

@ddilbazTT

Copy link
Copy Markdown
Contributor

Deepseek Qwen op no. 9552 fails with out-of-memory error. This is expected to be caught by try-except block. Instead, the whole program crashes.

The error is related to multiprocessing in Python. There is no full isolation between parent and child processes. So, the child process running out of memory kills the parent process.

Thus, now there is a memory limit to inputs that we pass on to the child process, and if exceeded, we push the inputs to disk and send a reference to disk address to child process. This is not the ideal solution, but it works using python's multiprocessing module.

  • helper functions such as get_tensor_size, get_inputs_size help determine size of inputs in bytes
  • run_op determines if inputs exceed 1 GB or not
  • if exceeding, inputs are dumped to a temporary file using pickle
  • execute_process uses pickle.load() if inputs exceed memory limit
  • file where inputs are stored is deleted at the end of run_op

As a note, PyTorch also faces a similar issue, and seems like they have extended torch.multiprocessing to handle large tensor sharing over processes.

Ticket

#291

@ddilbazTT

Copy link
Copy Markdown
Contributor Author

I have tested if deepseek_qwen would run on a nightly test and it successfully did: https://github.com/tenstorrent/tt-torch/actions/runs/13933783498

@ddilbazTT
ddilbazTT requested a review from LPanosTT March 19, 2025 14:37
Comment thread tt_torch/dynamo/executor.py Resolved
Comment thread tt_torch/dynamo/executor.py Resolved
Comment thread tt_torch/dynamo/executor.py Resolved
@github-actions

Copy link
Copy Markdown
TestsPassed ✅Skipped ⚠️Failed
TT-Torch Tests466 ran333 passed133 skipped0 failed
TestResult
No test annotations available

@codecov-commenter

codecov-commenter commented Mar 19, 2025 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 32.89474% with 51 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.15%. Comparing base (586dd1f) to head (3b342f9).
⚠️ Report is 519 commits behind head on main.

Files with missing lines Patch % Lines
tt_torch/dynamo/executor.py 32.89% 51 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #456      +/-   ##
==========================================
- Coverage   72.25%   70.15%   -2.10%     
==========================================
  Files          13       13              
  Lines        1820     1893      +73     
==========================================
+ Hits         1315     1328      +13     
- Misses        505      565      +60     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Comment thread tt_torch/dynamo/executor.py Resolved
Comment thread tt_torch/dynamo/executor.py Resolved
Comment thread tt_torch/dynamo/executor.py Resolved
Comment thread tt_torch/dynamo/executor.py Resolved
Comment thread tt_torch/dynamo/executor.py Resolved
Comment thread tt_torch/dynamo/executor.py Resolved
@github-actions

Copy link
Copy Markdown
TestsPassed ✅Skipped ⚠️Failed
TT-Torch Tests466 ran333 passed133 skipped0 failed
TestResult
No test annotations available

@github-actions

Copy link
Copy Markdown
TestsPassed ✅Skipped ⚠️Failed
TT-Torch Tests466 ran333 passed133 skipped0 failed
TestResult
No test annotations available

@AleksKnezevic

Copy link
Copy Markdown
Contributor

@jazpurTT, can you please check your model with this change too?

Deepseek Qwen op no. 9552 fails with out-of-memory error. This is
expected to be caught by try-except block. Instead,
the whole program crashes.

The error is related to multiprocessing in Python. There is no full
isolation between parent and child processes. So, the child process
running out of memory kills the parent process.

Thus, now there is a memory limit to inputs that we pass on to the child
process, and if exceeded, we push the inputs to disk and send a
reference to disk address to child process. This is not the ideal
solution, but it works using python's multiprocessing module.

- helper functions such as `get_tensor_size`, `get_inputs_size` help
  determine size of inputs in bytes
- `run_op` determines if inputs exceed 1 GB or not
- if exceeding, inputs are dumped to a temporary file using pickle
- `execute_process` uses `pickle.load()` if inputs exceed memory limit
- file where inputs are stored is deleted at the end of `run_op`

As a note, PyTorch also faces a similar issue, and seems like they have
extended torch.multiprocessing to handle large tensor sharing over
processes.
@jazpurTT

Copy link
Copy Markdown
Contributor

@jazpurTT, can you please check your model with this change too?

I did, it is compiling op_by_op, no more Bus Error. There is just an ATOL error during verify_output.

@ddilbazTT
ddilbazTT enabled auto-merge (squash) March 19, 2025 20:00
@github-actions

Copy link
Copy Markdown
TestsPassed ✅Skipped ⚠️Failed
TT-Torch Tests466 ran333 passed133 skipped0 failed
TestResult
No test annotations available

@ddilbazTT
ddilbazTT merged commit 8407a7f into main Mar 19, 2025
@ddilbazTT
ddilbazTT deleted the deepseek_qwen branch March 19, 2025 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants