You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
deepseek qwen fails in commit hash d27990d --> tt-mlir uplift on 2025-01-31
due to "Bus Error" --> this implies trying to access forbidden memory
Steps to reproduce:
you can single step through run_op to see where it happens. What I've done previously, to speed up debug, is conditionally compile/execute only the op I was interested in. than you can repro in a few seconds.
Activity
ddilbazTT commentedon Feb 18, 2025
deepseek qwen passes in commit hash 2256f17 --> tt-mlir uplift on 2025-01-25 on tt-torch
Steps to reproduce:
Branch:
deepseek_qwen_2256f17The tt-mlir commit hash corresponds to e581e6a815de6b0ce3c4ceedc70620d19be29969
deepseek qwen fails in commit hash d27990d --> tt-mlir uplift on 2025-01-31
due to "Bus Error" --> this implies trying to access forbidden memory
Steps to reproduce:
Branch:
deepseek_qwen_d27990ddilbazTT commentedon Feb 19, 2025
Debugged further. The error is happening when compiling node.idx == 9552. The following is some of the debug info:
The error happens during
calculated, runtime_stack_dump = self.run_op(binary, *args)AleksKnezevic commentedon Feb 19, 2025
you can single step through run_op to see where it happens. What I've done previously, to speed up debug, is conditionally compile/execute only the op I was interested in. than you can repro in a few seconds.
My guess is that it's in the json generation
ddilbazTT commentedon Feb 19, 2025
The error doesn't happens during compilation, it happens during execution. So I don't think it's json generation.
AleksKnezevic commentedon Feb 19, 2025
right, makes sense.
Fix Deepseek Qwen Fatal Bus Error [#291]
Fix Deepseek Qwen Fatal Bus Error [#291]
Fix Deepseek Qwen Fatal Bus Error [#291] (#456)