Skip to content

Deepseek qwen test fails in op-by-op #291

Closed

Description

@AleksKnezevic

Please debug and re-enable.

Activity

  1. ddilbazTT commented on Feb 18, 2025

    @ddilbazTT
    Contributor

    deepseek qwen passes in commit hash 2256f17 --> tt-mlir uplift on 2025-01-25 on tt-torch
    Steps to reproduce:

    git checkout -b deepseek_qwen_2256f17 2256f17
    deactivate; rm -rf env/venv build install third_party/tt-mlir/
    source env/activate
    cmake -G Ninja -B build
    cmake --build build
    cmake --install build
    pytest -s tests/models/deepseek/test_deepseek_qwen.py --op_by_op | tee 2256f17_deepseek_qwen.log
    
    

    Branch: deepseek_qwen_2256f17

    The tt-mlir commit hash corresponds to e581e6a815de6b0ce3c4ceedc70620d19be29969


    deepseek qwen fails in commit hash d27990d --> tt-mlir uplift on 2025-01-31
    due to "Bus Error" --> this implies trying to access forbidden memory
    Steps to reproduce:

    git checkout -b deepseek_qwen_d27990d d27990d
    deactivate; rm -rf env/venv build install third_party/tt-mlir/
    cmake -G Ninja -B build
    cmake --build build
    cmake --install build
    pytest -s tests/models/deepseek/test_deepseek_qwen.py --op_by_op 2>&1 | tee d27990d_deepseek_qwen.log
    
    

    Branch: deepseek_qwen_d27990

  2. ddilbazTT commented on Feb 19, 2025

    @ddilbazTT
    Contributor

    Debugged further. The error is happening when compiling node.idx == 9552. The following is some of the debug info:

    (Pdb) p op.stable_hlo_graph
    'module {\n  func.func @main(%arg0: tensor<10x5120xbf16>, %arg1: tensor<5120x152064xbf16>) -> tensor<10x152064xbf16> {\n    %0 = stablehlo.dot_general %arg0, %arg1, contracting_dims = [1
    ] x [0] : (tensor<10x5120xbf16>, tensor<5120x152064xbf16>) -> tensor<10x152064xbf16>\n    return %0 : tensor<10x152064xbf16>\n  }\n}\n'
    (Pdb) p op.torch_ir_graph
    'module {\n  func.func @main(%arg0: !torch.vtensor<[10,5120],bf16>, %arg1: !torch.vtensor<[5120,152064],bf16>) -> !torch.vtensor<[10,152064],bf16> {\n    %0 = torch.aten.mm %arg0, %arg1
    : !torch.vtensor<[10,5120],bf16>, !torch.vtensor<[5120,152064],bf16> -> !torch.vtensor<[10,152064],bf16>\n    return %0 : !torch.vtensor<[10,152064],bf16>\n  }\n}\n'
    
    

    The error happens during
    calculated, runtime_stack_dump = self.run_op(binary, *args)

  3. AleksKnezevic commented on Feb 19, 2025

    @AleksKnezevic
    ContributorAuthor

    you can single step through run_op to see where it happens. What I've done previously, to speed up debug, is conditionally compile/execute only the op I was interested in. than you can repro in a few seconds.

    My guess is that it's in the json generation

  4. ddilbazTT commented on Feb 19, 2025

    @ddilbazTT
    Contributor

    The error doesn't happens during compilation, it happens during execution. So I don't think it's json generation.

  5. AleksKnezevic commented on Feb 19, 2025

    @AleksKnezevic
    ContributorAuthor

    right, makes sense.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions