Skip to content

dynamic_dim error - The following dimensions have been specialized and CANNOT be dynamic #1379

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
adonnini opened this issue Dec 8, 2023 · 57 comments
Closed

dynamic_dim error - The following dimensions have been specialized and CANNOT be dynamic #1379

adonnini opened this issue Dec 8, 2023 · 57 comments
Assignees
@angelayi
Labels
actionable Items in the backlog waiting for an appropriate impl/fix module: exir Issues related to Export IR and the code under exir/ triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module

Comments

@adonnini
Copy link

adonnini commented Dec 8, 2023

I added dynamic_dim support to my model. Execution failed. Please find the full traceback log below.

My constrains are defined as follows:

        constraints = [
            dynamic_dim(enc_input_tensor, 0),
            dynamic_dim(dec_input, 0),
        ]

Is there a way to resolvew this error? How should I proceed?

Thanks

Traceback (most recent call last):
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/train-minimum.py", line 349, in <module>
    pre_autograd_aten_dialect = capture_pre_autograd_graph(m, (enc_input, dec_input, dec_source_mask, dec_target_mask), constraints=constraints)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py", line 328, in capture_pre_autograd_graph
    m = torch._dynamo.export(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 1252, in inner
    raise constraint_violation_error
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 1213, in inner
    result_traced = opt_f(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1519, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1528, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 401, in _fn
    return fn(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1519, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1528, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 549, in catch_errors
    return callback(frame, cache_entry, hooks, frame_state)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 142, in _fn
    return fn(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 384, in _convert_frame_assert
    return _compile(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 570, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 221, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 550, in compile_inner
    check_fn = CheckFunctionManager(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/guards.py", line 1015, in __init__
    guard.create(local_builder, global_builder)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_guards.py", line 259, in create
    return self.create_fn(self.source.select(local_builder, global_builder), self)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/guards.py", line 665, in SHAPE_ENV
    guards = output_graph.shape_env.produce_guards(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/fx/experimental/symbolic_shapes.py", line 3506, in produce_guards
    raise ConstraintViolationError(
torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['enc_input'].size()[0], L['dec_input'].size()[0])! For more information, run with TORCH_LOGS=dynamic.
  - Not all values of L['enc_input'].size()[0] = L['enc_input'].size()[0] in the specified range are valid because L['enc_input'].size()[0] was inferred to be a constant (27).
  - Not all values of L['dec_input'].size()[0] = L['dec_input'].size()[0] in the specified range are valid because L['dec_input'].size()[0] was inferred to be a constant (27).

The following dimensions have been specialized and CANNOT be dynamic.

def specializations(enc_input, dec_input, dec_source_mask, dec_target_mask):
# enc_input:
assert enc_input.size()[0] == 27
assert enc_input.size()[1] == 7
assert enc_input.size()[2] == 2

# dec_input:
assert dec_input.size()[0] == 27
assert dec_input.size()[1] == 12
assert dec_input.size()[2] == 3

# dec_source_mask:
assert dec_source_mask.size()[0] == 27
assert dec_source_mask.size()[1] == 1
assert dec_source_mask.size()[2] == 7

# dec_target_mask:
assert dec_target_mask.size()[0] == 27
assert dec_target_mask.size()[1] == 12
assert dec_target_mask.size()[2] == 12
@adonnini
Copy link
Author

adonnini commented Dec 9, 2023

@kimishpatel What makes executorch decide:
"The following dimensions have been specialized and CANNOT be dynamic."
?
What's the rationale and process used?
I see discrepancies with values defined in the model. For example:
dimension 0 is batch size which in the model is 64. How is 27, the number used/derived by executorch arrived at?

I tried to use dynamic_dim support and failed.

I will try disabling memory planning next. I am not optimistic.

What do you advise I do at this point? Should I simply sit tight and wait for (I am not sure what)?

I can' believe that the model I use is one-of-a-kind. In fact, it's a pretty mainstream model. It would be very useful to have a "cookbook" for addressing issues such as the ones I have encountewred unless you believe I ran into them because it's still early in the executorch development cycle.

Thanks for your help and patience

@kimishpatel
Copy link
Contributor

@adonnini

"The following dimensions have been specialized and CANNOT be dynamic."
?
THis is actually done by export and not executorch.

Journey of eager/python model to executorch includes

  • export (catpure_pre_autograd_graph)
  • to_edge
  • to_executorch

Failure you see is happening in the first step.

@angelayi can you take a look, please?

Another thing I suspect is how dec_source_mask and dec_target_mask need to also specify dynamic dim as it may related to dec_input. @angelayi can you help with this? If they are then maybe we need better error messages.

It would be very useful to have a "cookbook" for addressing issues such as the ones I have encountewred unless you believe I ran into them because it's still early in the executorch development cycle.

No this totally makes sense. SInce this is in export @angelayi do you know if issues of this nature hvae been addressed before. On executorch side, as we see issues of this nature, we need to add them to FAQ and redirect appropriaately.

@adonnini thanks for going through all this trouble. It is super helpful to us as well.

@SS-JIA SS-JIA added actionable Items in the backlog waiting for an appropriate impl/fix module: exir Issues related to Export IR and the code under exir/ triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels Dec 11, 2023
@larryliu0820
Copy link
Contributor

It will also be useful if you share your export() code. I'm curious what the input looks like.

@angelayi angelayi added need-user-input The issue needs more information from the reporter before moving forward and removed actionable Items in the backlog waiting for an appropriate impl/fix need-user-input The issue needs more information from the reporter before moving forward labels Dec 11, 2023
@adonnini
Copy link
Author

@ here is my executorch related code:

        ENCODER_IP_SIZE = encoder_ip_size
        DECODER_IP_SIZE = decoder_ip_size
        MODEL_OP_SIZE =  model_op_size
        EMB_SIZE = emb_size
        NUM_HEADS = num_heads
        FF_HIDDEN_SIZE = ff_hidden_size
        NUMBER_LAYERS = n
        DROPOUT = dropout
        ENC_INPUT = enc_input
        DEC_INPUT = dec_input
        DEC_SOURCE_MASK = dec_source_mask
        DEC_TARGET_MASK = dec_target_mask

        m = model.TFModel(ENCODER_IP_SIZE, DECODER_IP_SIZE, MODEL_OP_SIZE, EMB_SIZE, NUM_HEADS, FF_HIDDEN_SIZE, NUMBER_LAYERS, DROPOUT)

        m.eval();

        print(exir.capture(m, (enc_input, dec_input, dec_source_mask, dec_target_mask)).to_edge())

        constraints = [
            dynamic_dim(enc_input_tensor, 0),
            dynamic_dim(dec_input, 0),
        ]

        pre_autograd_aten_dialect = capture_pre_autograd_graph(m, (enc_input, dec_input, dec_source_mask, dec_target_mask), constraints=constraints)
        aten_dialect: ExportedProgram = export(pre_autograd_aten_dialect, (enc_input, dec_input, dec_source_mask, dec_target_mask))
        edge_program: exir.EdgeProgramManager = exir.to_edge(aten_dialect)
        executorch_program: exir.ExecutorchProgramManager = edge_program.to_executorch(
        ExecutorchBackendConfig(
        # passes=[],  # User-defined passes
        )
        )

        with open("/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/models/tfmodel.pte", "wb") as file:
            file.write(executorch_program.buffer)

        edge_program = edge_program.to_backend(XnnpackPartitioner)
        exec_prog = edge_program.to_executorch()

        with open("/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/models/tfmodel_exnnpack.pte", "wb") as file:
            file.write(exec_prog.buffer)

@adonnini
Copy link
Author

@kimishpatel thanks for your response.

At this point, you should assume that decoder input is dynamic.

Please let me know if you need additional information or you need me to do anything.

Thanks

@angelayi
Copy link
Contributor

@adonnini Kimish mentioned this before, but it seems like the shape of dec_input is dependent on or related to dec_source_mask and dec_target_mask, as the first dimension of those are the same shape. Can you try making those also dynamic?

@adonnini
Copy link
Author

@angelayi

dec_source_mask and dec_target_mask are dynamic.

I will double check to be sure. I will let you know if they are not.

Thanks.

@adonnini
Copy link
Author

@angelayi I double checked. dec_source_mask and dec_target_mask are dynamic.

@angelayi
Copy link
Contributor

angelayi commented Dec 12, 2023 •
edited
Loading

@adonnini did you change your constraints to be the following?

        constraints = [
            dynamic_dim(enc_input_tensor, 0),
            dynamic_dim(dec_input, 0),
            dynamic_dim(dec_source_mask, 0),
            dynamic_dim(dec_target_mask, 0),
        ]

@adonnini
Copy link
Author

@angelayi my constraints are currently:

        constraints = [
            dynamic_dim(enc_input_tensor, 0),
            dynamic_dim(dec_input, 0),
        ]

Do you want me to change them to:

        constraints = [
            dynamic_dim(enc_input_tensor, 0),
            dynamic_dim(dec_input, 0),
            dynamic_dim(dec_source_mask, 0),
            dynamic_dim(dec_target_mask, 0),
        ]

?

@adonnini
Copy link
Author

adonnini commented Dec 13, 2023 •
edited
Loading

@angelayi I changed the constraints as you suggested. Execution failed . the tracebac log included this:

The following dimensions CAN be dynamic.
Please use the following code to specify the constraints they must satisfy:

def specify_constraints(enc_input, dec_input, dec_source_mask, dec_target_mask):
    return [
        # dec_input:
        dynamic_dim(dec_input, 0) == dynamic_dim(enc_input, 0),

        # dec_source_mask:
        dynamic_dim(dec_source_mask, 0) == dynamic_dim(enc_input, 0),

        # dec_target_mask:
        dynamic_dim(dec_target_mask, 0) == dynamic_dim(enc_input, 0),
    ]

I made the change suggested above. Execution still failed. Below is the complete traceback log.

Please note that in my code passes[] is commented out:

        ExecutorchBackendConfig(
        # passes=[],  # User-defined passes
        )
        )

Traceback (most recent call last):
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/fx/passes/infra/pass_manager.py", line 270, in __call__
    res = fn(module)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/fx/passes/infra/pass_base.py", line 40, in __call__
    res = self.call(graph_module)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/passes/add_runtime_assertions_for_constraints_pass.py", line 142, in call
    val = super().call(graph_module)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/pass_base.py", line 424, in call
    result = self.call_submodule(graph_module, tuple(inputs))
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/pass_base.py", line 388, in call_submodule
    interpreter.run(*inputs_data)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/fx/interpreter.py", line 138, in run
    self.env[node] = self.run_node(node)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/pass_base.py", line 232, in run_node
    return super().run_node(n)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/fx/interpreter.py", line 195, in run_node
    return getattr(self, n.op)(n.target, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/pass_base.py", line 209, in call_function
    raise ExportPassBaseError(f"Unsupported target type: {target}")
torch._export.pass_base.ExportPassBaseError: Unsupported target type: <function sym_sqrt at 0x7f4ce20394c0>

While executing %sym_sqrt : [num_users=1] = call_function[target=torch.fx.experimental.symbolic_shapes.sym_sqrt](args = (%sym_size,), kwargs = {})
Original traceback:
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py", line 487, in forward
    encoder_output = self.encoder_block.forward(enc_embed)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py", line 193, in forward
    x = layer.forward(x)  # Shape = (B, N, C)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py", line 147, in forward
    self.attn.forward(self.norm_attn(x), self.norm_attn(x), self.norm_attn(x)))  # Shape = (B, N ,C)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py", line 87, in forward
    attn_output = attention(Q, K, V, mask, self.dropout)  # Shape = (B, H, N, C//H)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/utils.py", line 49, in attention
    scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(new_emb_size)


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/train-minimum.py", line 381, in <module>
    aten_dialect: ExportedProgram = export(pre_autograd_aten_dialect, (enc_input, dec_input, dec_source_mask, dec_target_mask))
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/export/__init__.py", line 586, in export
    return export__RC__(f, args, kwargs, dynamic_shapes=dynamic_shapes)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py", line 95, in export__RC__
    return export(f, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py", line 470, in export
    return _export(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py", line 755, in _export
    exported_program = exported_program._transform(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/export/exported_program.py", line 569, in _transform
    res = pm(self.graph_module)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/fx/passes/infra/pass_manager.py", line 296, in __call__
    raise Exception(msg) from e
Exception: An error occurred when running the '_AddRuntimeAssertionsForInlineConstraintsPass' pass after the following passes: []

@angelayi
Copy link
Contributor

angelayi commented Dec 13, 2023 •
edited
Loading

I can send out a fix for that on the export side, but I don't think executorch can handle those ops either. cc @larryliu0820

@adonnini
Copy link
Author

@angelayi where/when could I get the fix?
@larryliu0820 does executorch also have the same problem?
Thanks

@angelayi angelayi mentioned this issue Dec 14, 2023
Closed
angelayi added a commit to pytorch/pytorch that referenced this issue Dec 14, 2023
@angelayi
[export] Support torch.sym* ops
3805d3c 
Fixes #108830
pytorch/executorch#1379 (comment)
@adonnini
Copy link
Author

@angelayi Thanks. I know it may not be the "best practice" way of doing things, but could I simply replace on my system the modules you changed with your commit with the updated ones?
Thanks

@angelayi
Copy link
Contributor

@adonnini I'm not sure what you mean, but if it works for you then it should be fine!

@SS-JIA SS-JIA added the actionable Items in the backlog waiting for an appropriate impl/fix label Dec 15, 2023
pytorchmergebot pushed a commit to pytorch/pytorch that referenced this issue Dec 15, 2023
@angelayi @pytorchmergebot
[export] Support torch.sym* ops (#115854)
347cb91 
Fixes #108830 and pytorch/executorch#1379 (comment)

Pull Request resolved: #115854
Approved by: https://github.com/zhxchen17
@adonnini
Copy link
Author

@angelayi my simplistic approach to using the updated torch.export was completely off-base.. The fixes don't have anything to do with executorch per se. I forgot that key point.
I would think that your fixes are not yet part of the official release of torch._export and torch.export, are they?
If they are not, how can I access the version of torch.export and torch._export with your fixes?
Thanks and sorry for the confusion.

guilhermeleobas pushed a commit to guilhermeleobas/pytorch that referenced this issue Dec 18, 2023
@angelayi @guilhermeleobas
[export] Support torch.sym* ops (pytorch#115854)
b82f743 
Fixes pytorch#108830 and pytorch/executorch#1379 (comment)

Pull Request resolved: pytorch#115854
Approved by: https://github.com/zhxchen17
pytorchmergebot pushed a commit to pytorch/pytorch that referenced this issue Dec 18, 2023
@angelayi @pytorchmergebot
[export] Support torch.sym* ops (#115854)
e43d33f 
Fixes #108830 and pytorch/executorch#1379 (comment)

Pull Request resolved: #115854
Approved by: https://github.com/zhxchen17
dmenig pushed a commit to dmenig/pytorch that referenced this issue Dec 21, 2023
@angelayi @dmenig
[export] Support torch.sym* ops (pytorch#115854)
5a48cff 
Fixes pytorch#108830 and pytorch/executorch#1379 (comment)

Pull Request resolved: pytorch#115854
Approved by: https://github.com/zhxchen17
@adonnini
Copy link
Author

@angelayi I commented out import torchvision in the trainer as it is not used (because I commented out other code that depended on it). Now the trainer runs and fails producing the traceback log reported below.

Please let me know if/what I should do to help address this issue. Thanks

Traceback (most recent call last):
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/train-minimum.py", line 381, in <module>
    aten_dialect: ExportedProgram = export(pre_autograd_aten_dialect, (enc_input, dec_input, dec_source_mask, dec_target_mask))
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/export/__init__.py", line 187, in export
    return _export(
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/export/exported_program.py", line 77, in wrapper
    return fn(*args, **kwargs)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/export/_trace.py", line 559, in _export
    gm_torch_level = _export_to_torch_ir(
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/export/_trace.py", line 261, in _export_to_torch_ir
    gm_torch_level, _ = torch._dynamo.export(
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 1277, in inner
    result_traced = opt_f(*args, **kwargs)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 434, in _fn
    return fn(*args, **kwargs)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/fx/graph_module.py", line 738, in call_wrapped
    return self._wrapped_call(self, *args, **kwargs)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/fx/graph_module.py", line 317, in __call__
    raise e
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/fx/graph_module.py", line 304, in __call__
    return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1561, in _call_impl
    result = forward_call(*args, **kwargs)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 597, in catch_errors
    return callback(frame, cache_entry, hooks, frame_state)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 382, in _convert_frame_assert
    return _compile(
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 639, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 246, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 520, in compile_inner
    out_code = transform_code_object(code, transform)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/_dynamo/bytecode_transformation.py", line 1033, in transform_code_object
    transformations(instructions, code_options)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 151, in _fn
    return fn(*args, **kwargs)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 485, in transform
    tracer.run()
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 2096, in run
    super().run()
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 781, in run
    and self.step()
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 744, in step
    getattr(self, inst.opname)(inst)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 463, in wrapper
    return inner_fn(self, inst)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 1176, in CALL_FUNCTION
    self.call_function(fn, args, {})
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 645, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/_dynamo/variables/torch.py", line 738, in call_function
    tensor_variable = wrap_fx_proxy(
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/_dynamo/variables/builder.py", line 1326, in wrap_fx_proxy
    return wrap_fx_proxy_cls(target_cls=TensorVariable, **kwargs)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/_dynamo/variables/builder.py", line 1581, in wrap_fx_proxy_cls
    unimplemented(
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/.executorch/lib/python3.9/site-packages/torch/_dynamo/exc.py", line 193, in unimplemented
    raise Unsupported(msg)
torch._dynamo.exc.Unsupported: torch.* op returned non-Tensor float call_function <function sym_sqrt at 0x7f280ff601f0>

from user code:
   File "<eval_with_key>.238", line 47, in forward
    sym_sqrt = torch.sym_sqrt(sym_size_int)

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information

@adonnini
Copy link
Author

@angelayi I tried using the 12/22 nightly build in case the bug had been fixed. The result was the same

@adonnini
Copy link
Author

adonnini commented Jan 9, 2024

Hi @angelayi can you suggest a next step I could take to resolve the problem? I am a bit stuck and cannot proceed with some of my work until I resolve this issue. Thanks

@angelayi
Copy link
Contributor

angelayi commented Jan 9, 2024

@adonnini

Can you update everything to the latest nightlies?
And then run the model with your inputs eagerly to check that that your modified model is running correctly?
And then trace the function using torch.export.export(...) without using capture_pre_autograd?
And then if that still fails run it again with TORCH_LOGS=dynamo,dynamic and please send the logs over.

@adonnini
Copy link
Author

@angelayi Thanks. I am in the process of following your instructions. Completed steps 1 and 2.
I am a bit puzzled by your asking me to run "modified model". I did not modify my model except for adding executorch related code to the training mofdule.
What are you referring to when you say "modified model"?
Thanks

@angelayi
Copy link
Contributor

@adonnini You mentioned that you "commented out import torchvision in the trainer as it is not used (because I commented out other code that depended on it)". I thought that meant that you modified model code?

@adonnini
Copy link
Author

@angelayi Ah, I see. Sorry. OK. I did run the modified model. On to steps 3 and 4

@adonnini
Copy link
Author

@angelayi unfortunately code execution still fails "without using capture_pre_autograd".

Below you will find:

  1. the latest executorch related code I use in my train module
  2. Traceback logs produced with
        torch._logging.set_logs(dynamo = logging.DEBUG)
        torch._dynamo.config.verbose = True
  1. Traceback log produced by setting TORCH_LOGS=dynamo,dynamic

Please let me know if you need anything else or if I should do something else.

Thanks

CODE

        ENCODER_IP_SIZE = encoder_ip_size
        DECODER_IP_SIZE = decoder_ip_size
        MODEL_OP_SIZE =  model_op_size
        EMB_SIZE = emb_size
        NUM_HEADS = num_heads
        FF_HIDDEN_SIZE = ff_hidden_size
        NUMBER_LAYERS = n
        DROPOUT = dropout
        ENC_INPUT = enc_input
        DEC_INPUT = dec_input
        DEC_SOURCE_MASK = dec_source_mask
        DEC_TARGET_MASK = dec_target_mask

        m = model.TFModel(ENCODER_IP_SIZE, DECODER_IP_SIZE, MODEL_OP_SIZE, EMB_SIZE, NUM_HEADS, FF_HIDDEN_SIZE, NUMBER_LAYERS, DROPOUT)

        m.eval();

        constraints = [
            dynamic_dim(enc_input_tensor, 0),
            dynamic_dim(dec_input, 0),
            dynamic_dim(dec_source_mask, 0),
            dynamic_dim(dec_target_mask, 0),
        ]
 
        # pre_autograd_aten_dialect = capture_pre_autograd_graph(m, (enc_input, dec_input, dec_source_mask, dec_target_mask), constraints=constraints)
        aten_dialect: ExportedProgram = export(m, (enc_input, dec_input, dec_source_mask, dec_target_mask))
        # aten_dialect: ExportedProgram = export(pre_autograd_aten_dialect, (enc_input, dec_input, dec_source_mask, dec_target_mask))
        edge_program: exir.EdgeProgramManager = exir.to_edge(aten_dialect)
        executorch_program: exir.ExecutorchProgramManager = edge_program.to_executorch(
        ExecutorchBackendConfig(
        # passes=[],  # User-defined passes
        )
        )

        with open("/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/models/tfmodel.pte", "wb") as file:
            file.write(executorch_program.buffer)

        edge_program = edge_program.to_backend(XnnpackPartitioner)
        exec_prog = edge_program.to_executorch()

        with open("/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/models/tfmodel_exnnpack.pte", "wb") as file:
            file.write(exec_prog.buffer)

TRACEBACK LOG PRODUCED USING

        torch._logging.set_logs(dynamo = logging.DEBUG)
        torch._dynamo.config.verbose = True

(base) adonnini1@actlnxlptp6:~/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master$ python3 train-minimum.py
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py:186: UserWarning: This function is deprecated. Please use torch.export.export instead.
  warnings.warn("This function is deprecated. Please use torch.export.export instead.")
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py:186: UserWarning: This function is deprecated. Please use torch.export.export instead.
  warnings.warn("This function is deprecated. Please use torch.export.export instead.")
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py:186: UserWarning: This function is deprecated. Please use torch.export.export instead.
  warnings.warn("This function is deprecated. Please use torch.export.export instead.")
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py:186: UserWarning: This function is deprecated. Please use torch.export.export instead.
  warnings.warn("This function is deprecated. Please use torch.export.export instead.")
Value of eta max is: 0.0001
  0%|                                                                                                                                                  | 0/5 [00:00<?, ?it/s]Epoch 1/5....Training loss = 8.9662
[2024-01-09 20:02:45,116] [4/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py:471
[2024-01-09 20:02:45,116] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2024-01-09 20:02:45,117] [4/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py:471 in forward (TFModel.forward)
[2024-01-09 20:02:45,117] [4/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]         def forward(self, enc_input, dec_input, dec_source_mask, dec_target_mask):
[2024-01-09 20:02:45,126] [4/0] torch._dynamo.output_graph: [DEBUG] create_graph_input L_enc_input_ L['enc_input']
[2024-01-09 20:02:45,127] [4/0] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['enc_input'] (27, 7, 2) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>], constraint_sizes=[None, None, None], tensor_source=LocalSource(local_name='enc_input', cell_or_freevar=False), shape_env_to_source_to_symbol_cache={})
[2024-01-09 20:02:45,127] [4/0] torch._dynamo.output_graph: [DEBUG] create_graph_input L_dec_input_ L['dec_input']
[2024-01-09 20:02:45,127] [4/0] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['dec_input'] (27, 12, 3) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>], constraint_sizes=[None, None, None], tensor_source=LocalSource(local_name='dec_input', cell_or_freevar=False), shape_env_to_source_to_symbol_cache={})
[2024-01-09 20:02:45,128] [4/0] torch._dynamo.output_graph: [DEBUG] create_graph_input L_dec_source_mask_ L['dec_source_mask']
[2024-01-09 20:02:45,128] [4/0] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['dec_source_mask'] (27, 1, 7) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>], constraint_sizes=[None, None, None], tensor_source=LocalSource(local_name='dec_source_mask', cell_or_freevar=False), shape_env_to_source_to_symbol_cache={})
[2024-01-09 20:02:45,128] [4/0] torch._dynamo.output_graph: [DEBUG] create_graph_input L_dec_target_mask_ L['dec_target_mask']
[2024-01-09 20:02:45,128] [4/0] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['dec_target_mask'] (27, 12, 12) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>], constraint_sizes=[None, None, None], tensor_source=LocalSource(local_name='dec_target_mask', cell_or_freevar=False), shape_env_to_source_to_symbol_cache={})
[2024-01-09 20:02:45,129] [4/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py:486 in forward (TFModel.forward)
[2024-01-09 20:02:45,129] [4/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]             enc_embed = self.encoder_embedding.forward(enc_input)
[2024-01-09 20:02:45,129] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2024-01-09 20:02:45,129] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR encoder_embedding [NNModuleVariable()]
[2024-01-09 20:02:45,129] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2024-01-09 20:02:45,129] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST enc_input [UserMethodVariable(<function Embeddings.forward at 0x7fa616a96160>, NNModuleVariable())]
[2024-01-09 20:02:45,129] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [UserMethodVariable(<function Embeddings.forward at 0x7fa616a96160>, NNModuleVariable()), TensorVariable()]
[2024-01-09 20:02:45,129] [4/0] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7fa616a92920, file "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py", line 383>, inlined by default
[2024-01-09 20:02:45,130] [4/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py:383 in forward (Embeddings.forward) (inline depth: 1)
[2024-01-09 20:02:45,130] [4/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]         def forward(self, x):
[2024-01-09 20:02:45,130] [4/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py:395 in forward (Embeddings.forward) (inline depth: 1)
[2024-01-09 20:02:45,130] [4/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]             x = self.linear_embd(x) * math.sqrt(self.emb_size)  # Shape = (B, N, C)
[2024-01-09 20:02:45,130] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2024-01-09 20:02:45,130] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR linear_embd [NNModuleVariable()]
[2024-01-09 20:02:45,130] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [NNModuleVariable()]
[2024-01-09 20:02:45,130] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [NNModuleVariable(), TensorVariable()]
[2024-01-09 20:02:45,134] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL math [TensorVariable()]
[2024-01-09 20:02:45,134] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR sqrt [TensorVariable(), PythonModuleVariable(<module 'math' from '/home/adonnini1/anaconda3/lib/python3.9/lib-dynload/math.cpython-39-x86_64-linux-gnu.so'>)]
[2024-01-09 20:02:45,134] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [TensorVariable(), TorchInGraphFunctionVariable(<built-in function sqrt>)]
[2024-01-09 20:02:45,134] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR emb_size [TensorVariable(), TorchInGraphFunctionVariable(<built-in function sqrt>), NNModuleVariable()]
[2024-01-09 20:02:45,134] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TensorVariable(), TorchInGraphFunctionVariable(<built-in function sqrt>), ConstantVariable(int)]
[2024-01-09 20:02:45,134] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_MULTIPLY None [TensorVariable(), ConstantVariable(float)]
[2024-01-09 20:02:45,135] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE STORE_FAST x [TensorVariable()]
[2024-01-09 20:02:45,135] [4/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py:397 in forward (Embeddings.forward) (inline depth: 1)
[2024-01-09 20:02:45,135] [4/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]             x = self.pos_encoding.forward(x)
[2024-01-09 20:02:45,135] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self []
[2024-01-09 20:02:45,135] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR pos_encoding [NNModuleVariable()]
[2024-01-09 20:02:45,135] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR forward [NNModuleVariable()]
[2024-01-09 20:02:45,135] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [UserMethodVariable(<function PositionalEncoding.forward at 0x7fa616a96040>, NNModuleVariable())]
[2024-01-09 20:02:45,135] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [UserMethodVariable(<function PositionalEncoding.forward at 0x7fa616a96040>, NNModuleVariable()), TensorVariable()]
[2024-01-09 20:02:45,135] [4/0] torch._dynamo.symbolic_convert: [DEBUG] INLINING <code object forward at 0x7fa616a92710, file "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py", line 340>, inlined by default
[2024-01-09 20:02:45,136] [4/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py:340 in forward (PositionalEncoding.forward) (inline depth: 2)
[2024-01-09 20:02:45,136] [4/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]         def forward(self, x):
[2024-01-09 20:02:45,136] [4/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG] TRACE starts_line /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py:353 in forward (PositionalEncoding.forward) (inline depth: 2)
[2024-01-09 20:02:45,136] [4/0] torch._dynamo.symbolic_convert.__trace_source: [DEBUG]             x = x + Variable(self.pe[:, :x.size(1)], requires_grad=False)
[2024-01-09 20:02:45,136] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x []
[2024-01-09 20:02:45,136] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_GLOBAL Variable [TensorVariable()]
[2024-01-09 20:02:45,136] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST self [TensorVariable(), UserDefinedClassVariable(<class 'torch.autograd.variable.Variable'>)]
[2024-01-09 20:02:45,136] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR pe [TensorVariable(), UserDefinedClassVariable(<class 'torch.autograd.variable.Variable'>), NNModuleVariable()]
[2024-01-09 20:02:45,137] [4/0] torch._dynamo.variables.builder: [DEBUG] wrap_to_fake L['self'].encoder_embedding.pos_encoding.pe (1, 5000, 512) StatefulSymbolicContext(dynamic_sizes=[<DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>, <DimDynamic.STATIC: 2>], constraint_sizes=[None, None, None], tensor_source=NNModuleSource(base=AttrSource(base=NNModuleSource(base=AttrSource(base=NNModuleSource(base=AttrSource(base=LocalSource(local_name='self', cell_or_freevar=False), member='encoder_embedding')), member='pos_encoding')), member='pe')), shape_env_to_source_to_symbol_cache={})
[2024-01-09 20:02:45,138] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [TensorVariable(), UserDefinedClassVariable(<class 'torch.autograd.variable.Variable'>), TensorVariable()]
[2024-01-09 20:02:45,138] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [TensorVariable(), UserDefinedClassVariable(<class 'torch.autograd.variable.Variable'>), TensorVariable(), ConstantVariable(NoneType)]
[2024-01-09 20:02:45,138] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_SLICE 2 [TensorVariable(), UserDefinedClassVariable(<class 'torch.autograd.variable.Variable'>), TensorVariable(), ConstantVariable(NoneType), ConstantVariable(NoneType)]
[2024-01-09 20:02:45,138] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST None [TensorVariable(), UserDefinedClassVariable(<class 'torch.autograd.variable.Variable'>), TensorVariable(), SliceVariable()]
[2024-01-09 20:02:45,139] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_FAST x [TensorVariable(), UserDefinedClassVariable(<class 'torch.autograd.variable.Variable'>), TensorVariable(), SliceVariable(), ConstantVariable(NoneType)]
[2024-01-09 20:02:45,139] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_ATTR size [TensorVariable(), UserDefinedClassVariable(<class 'torch.autograd.variable.Variable'>), TensorVariable(), SliceVariable(), ConstantVariable(NoneType), TensorVariable()]
[2024-01-09 20:02:45,139] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST 1 [TensorVariable(), UserDefinedClassVariable(<class 'torch.autograd.variable.Variable'>), TensorVariable(), SliceVariable(), ConstantVariable(NoneType), GetAttrVariable(TensorVariable(), size)]
[2024-01-09 20:02:45,139] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION 1 [TensorVariable(), UserDefinedClassVariable(<class 'torch.autograd.variable.Variable'>), TensorVariable(), SliceVariable(), ConstantVariable(NoneType), GetAttrVariable(TensorVariable(), size), ConstantVariable(int)]
[2024-01-09 20:02:45,139] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_SLICE 2 [TensorVariable(), UserDefinedClassVariable(<class 'torch.autograd.variable.Variable'>), TensorVariable(), SliceVariable(), ConstantVariable(NoneType), ConstantVariable(int)]
[2024-01-09 20:02:45,139] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE BUILD_TUPLE 2 [TensorVariable(), UserDefinedClassVariable(<class 'torch.autograd.variable.Variable'>), TensorVariable(), SliceVariable(), SliceVariable()]
[2024-01-09 20:02:45,139] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE BINARY_SUBSCR None [TensorVariable(), UserDefinedClassVariable(<class 'torch.autograd.variable.Variable'>), TensorVariable(), TupleVariable()]
[2024-01-09 20:02:45,140] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST False [TensorVariable(), UserDefinedClassVariable(<class 'torch.autograd.variable.Variable'>), TensorVariable()]
[2024-01-09 20:02:45,140] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE LOAD_CONST ('requires_grad',) [TensorVariable(), UserDefinedClassVariable(<class 'torch.autograd.variable.Variable'>), TensorVariable(), ConstantVariable(bool)]
[2024-01-09 20:02:45,140] [4/0] torch._dynamo.symbolic_convert: [DEBUG] TRACE CALL_FUNCTION_KW 2 [TensorVariable(), UserDefinedClassVariable(<class 'torch.autograd.variable.Variable'>), TensorVariable(), ConstantVariable(bool), TupleVariable()]
[2024-01-09 20:02:45,140] [4/0] torch._dynamo.symbolic_convert: [DEBUG] empty checkpoint
[2024-01-09 20:02:45,140] [4/0] torch._dynamo.symbolic_convert: [DEBUG] FAILED INLINING <code object forward at 0x7fa616a92710, file "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py", line 340>
[2024-01-09 20:02:45,140] [4/0] torch._dynamo.symbolic_convert: [DEBUG] empty checkpoint
[2024-01-09 20:02:45,140] [4/0] torch._dynamo.symbolic_convert: [DEBUG] FAILED INLINING <code object forward at 0x7fa616a92920, file "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py", line 383>
[2024-01-09 20:02:45,140] [4/0] torch._dynamo.symbolic_convert: [DEBUG] empty checkpoint
  0%|                                                                                                                                                  | 0/5 [00:23<?, ?it/s]
Traceback (most recent call last):
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/train-minimum.py", line 404, in <module>
    aten_dialect: ExportedProgram = export(m, (enc_input, dec_input, dec_source_mask, dec_target_mask))
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/export/__init__.py", line 191, in export
    return _export(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/export/exported_program.py", line 78, in wrapper
    return fn(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/export/_trace.py", line 588, in _export
    gm_torch_level = _export_to_torch_ir(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/export/_trace.py", line 265, in _export_to_torch_ir
    gm_torch_level, _ = torch._dynamo.export(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 1268, in inner
    result_traced = opt_f(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 417, in _fn
    return fn(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 580, in catch_errors
    return callback(frame, cache_entry, hooks, frame_state)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 384, in _convert_frame_assert
    return _compile(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 643, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 247, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 524, in compile_inner
    out_code = transform_code_object(code, transform)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/bytecode_transformation.py", line 1033, in transform_code_object
    transformations(instructions, code_options)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 151, in _fn
    return fn(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 489, in transform
    tracer.run()
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 2098, in run
    super().run()
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 780, in run
    and self.step()
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 743, in step
    getattr(self, inst.opname)(inst)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 462, in wrapper
    return inner_fn(self, inst)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 1190, in CALL_FUNCTION
    self.call_function(fn, args, {})
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 644, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/variables/functions.py", line 301, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/variables/functions.py", line 255, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/variables/functions.py", line 84, in call_function
    return tx.inline_user_function_return(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 650, in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 2233, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 2348, in inline_call_
    tracer.run()
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 780, in run
    and self.step()
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 743, in step
    getattr(self, inst.opname)(inst)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 462, in wrapper
    return inner_fn(self, inst)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 1190, in CALL_FUNCTION
    self.call_function(fn, args, {})
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 644, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/variables/functions.py", line 301, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/variables/functions.py", line 255, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/variables/functions.py", line 84, in call_function
    return tx.inline_user_function_return(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 650, in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 2233, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 2348, in inline_call_
    tracer.run()
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 780, in run
    and self.step()
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 743, in step
    getattr(self, inst.opname)(inst)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 462, in wrapper
    return inner_fn(self, inst)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 1243, in CALL_FUNCTION_KW
    self.call_function(fn, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 644, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/variables/user_defined.py", line 398, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/variables/base.py", line 321, in call_function
    unimplemented(f"call_function {self} {args} {kwargs}")
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/exc.py", line 193, in unimplemented
    raise Unsupported(msg)
torch._dynamo.exc.Unsupported: call_function UserDefinedClassVariable() [TensorVariable()] {'requires_grad': ConstantVariable(bool)}

from user code:
   File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py", line 486, in forward
    enc_embed = self.encoder_embedding.forward(enc_input)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py", line 397, in forward
    x = self.pos_encoding.forward(x)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py", line 353, in forward
    x = x + Variable(self.pe[:, :x.size(1)], requires_grad=False)

[2024-01-09 20:02:45,146] torch._dynamo.utils: [INFO] TorchDynamo compilation metrics:
[2024-01-09 20:02:45,146] torch._dynamo.utils: [INFO] Function                           Runtimes (s)
[2024-01-09 20:02:45,146] torch._dynamo.utils: [INFO] -------------------------------  --------------
[2024-01-09 20:02:45,146] torch._dynamo.utils: [INFO] _compile.<locals>.compile_inner          0.1504
[2024-01-09 20:02:45,146] torch._dynamo.utils: [INFO] OutputGraph.call_user_compiler           0.0002
[2024-01-09 20:02:45,146] torch._dynamo.utils: [INFO] create_aot_dispatcher_function           0.3582
(base) adonnini1@actlnxlptp6:~/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master$

TRACEBACK LOG PRODUCED SETTING TORCH_LOGS=dynamo,dynamic


(base) adonnini1@actlnxlptp6:~/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master$ export TORCH_LOGS=dynamo,dynamic
(base) adonnini1@actlnxlptp6:~/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master$ python3 train-minimum.py
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py:186: UserWarning: This function is deprecated. Please use torch.export.export instead.
  warnings.warn("This function is deprecated. Please use torch.export.export instead.")
[2024-01-09 20:58:32,827] [0/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/backends/xnnpack/partition/graphs/bilinear_2d.py:19
[2024-01-09 20:58:32,827] [0/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2024-01-09 20:58:32,916] [0/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2024-01-09 20:58:32,917] [0/0] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function dynamo_normalization_capturing_compiler
[2024-01-09 20:58:32,917] [0/0] torch._dynamo.output_graph: [INFO] Step 2: done compiler function dynamo_normalization_capturing_compiler
[2024-01-09 20:58:32,918] [0/0] torch.fx.experimental.symbolic_shapes: [INFO] produce_guards
[2024-01-09 20:58:32,934] torch._dynamo.eval_frame: [INFO] Summary of dimension constraints:
[2024-01-09 20:58:32,934] torch._dynamo.eval_frame: [INFO] The following dimensions have been specialized and CANNOT be dynamic.
[2024-01-09 20:58:32,934] torch._dynamo.eval_frame: [INFO] ```
[2024-01-09 20:58:32,934] torch._dynamo.eval_frame: [INFO] def specializations(x):
[2024-01-09 20:58:32,934] torch._dynamo.eval_frame: [INFO]     # x:
[2024-01-09 20:58:32,934] torch._dynamo.eval_frame: [INFO]     assert x.size()[0] == 1
[2024-01-09 20:58:32,934] torch._dynamo.eval_frame: [INFO]     assert x.size()[1] == 3
[2024-01-09 20:58:32,934] torch._dynamo.eval_frame: [INFO]     assert x.size()[2] == 4
[2024-01-09 20:58:32,934] torch._dynamo.eval_frame: [INFO]     assert x.size()[3] == 4
[2024-01-09 20:58:32,934] torch._dynamo.eval_frame: [INFO] ```
[2024-01-09 20:58:32,934] torch._dynamo.eval_frame: [INFO] 
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py:186: UserWarning: This function is deprecated. Please use torch.export.export instead.
  warnings.warn("This function is deprecated. Please use torch.export.export instead.")
[2024-01-09 20:58:33,195] [1/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/backends/xnnpack/partition/graphs/bilinear_2d.py:19
[2024-01-09 20:58:33,195] [1/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2024-01-09 20:58:33,205] [1/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2024-01-09 20:58:33,206] [1/0] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function dynamo_normalization_capturing_compiler
[2024-01-09 20:58:33,206] [1/0] torch._dynamo.output_graph: [INFO] Step 2: done compiler function dynamo_normalization_capturing_compiler
[2024-01-09 20:58:33,207] [1/0] torch.fx.experimental.symbolic_shapes: [INFO] produce_guards
[2024-01-09 20:58:33,220] torch._dynamo.eval_frame: [INFO] Summary of dimension constraints:
[2024-01-09 20:58:33,220] torch._dynamo.eval_frame: [INFO] The following dimensions have been specialized and CANNOT be dynamic.
[2024-01-09 20:58:33,220] torch._dynamo.eval_frame: [INFO] ```
[2024-01-09 20:58:33,220] torch._dynamo.eval_frame: [INFO] def specializations(x):
[2024-01-09 20:58:33,220] torch._dynamo.eval_frame: [INFO]     # x:
[2024-01-09 20:58:33,220] torch._dynamo.eval_frame: [INFO]     assert x.size()[0] == 1
[2024-01-09 20:58:33,220] torch._dynamo.eval_frame: [INFO]     assert x.size()[1] == 3
[2024-01-09 20:58:33,220] torch._dynamo.eval_frame: [INFO]     assert x.size()[2] == 4
[2024-01-09 20:58:33,220] torch._dynamo.eval_frame: [INFO]     assert x.size()[3] == 4
[2024-01-09 20:58:33,220] torch._dynamo.eval_frame: [INFO] ```
[2024-01-09 20:58:33,220] torch._dynamo.eval_frame: [INFO] 
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py:186: UserWarning: This function is deprecated. Please use torch.export.export instead.
  warnings.warn("This function is deprecated. Please use torch.export.export instead.")
[2024-01-09 20:58:33,467] [2/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/backends/xnnpack/partition/graphs/bilinear_2d.py:19
[2024-01-09 20:58:33,467] [2/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2024-01-09 20:58:33,478] [2/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2024-01-09 20:58:33,479] [2/0] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function dynamo_normalization_capturing_compiler
[2024-01-09 20:58:33,479] [2/0] torch._dynamo.output_graph: [INFO] Step 2: done compiler function dynamo_normalization_capturing_compiler
[2024-01-09 20:58:33,480] [2/0] torch.fx.experimental.symbolic_shapes: [INFO] produce_guards
[2024-01-09 20:58:33,493] torch._dynamo.eval_frame: [INFO] Summary of dimension constraints:
[2024-01-09 20:58:33,493] torch._dynamo.eval_frame: [INFO] The following dimensions have been specialized and CANNOT be dynamic.
[2024-01-09 20:58:33,493] torch._dynamo.eval_frame: [INFO] ```
[2024-01-09 20:58:33,493] torch._dynamo.eval_frame: [INFO] def specializations(x):
[2024-01-09 20:58:33,493] torch._dynamo.eval_frame: [INFO]     # x:
[2024-01-09 20:58:33,493] torch._dynamo.eval_frame: [INFO]     assert x.size()[0] == 1
[2024-01-09 20:58:33,493] torch._dynamo.eval_frame: [INFO]     assert x.size()[1] == 3
[2024-01-09 20:58:33,493] torch._dynamo.eval_frame: [INFO]     assert x.size()[2] == 4
[2024-01-09 20:58:33,493] torch._dynamo.eval_frame: [INFO]     assert x.size()[3] == 4
[2024-01-09 20:58:33,493] torch._dynamo.eval_frame: [INFO] ```
[2024-01-09 20:58:33,493] torch._dynamo.eval_frame: [INFO] 
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py:186: UserWarning: This function is deprecated. Please use torch.export.export instead.
  warnings.warn("This function is deprecated. Please use torch.export.export instead.")
[2024-01-09 20:58:33,825] [3/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/backends/xnnpack/partition/graphs/bilinear_2d.py:19
[2024-01-09 20:58:33,825] [3/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2024-01-09 20:58:33,836] [3/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2024-01-09 20:58:33,837] [3/0] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function dynamo_normalization_capturing_compiler
[2024-01-09 20:58:33,837] [3/0] torch._dynamo.output_graph: [INFO] Step 2: done compiler function dynamo_normalization_capturing_compiler
[2024-01-09 20:58:33,838] [3/0] torch.fx.experimental.symbolic_shapes: [INFO] produce_guards
[2024-01-09 20:58:33,852] torch._dynamo.eval_frame: [INFO] Summary of dimension constraints:
[2024-01-09 20:58:33,852] torch._dynamo.eval_frame: [INFO] The following dimensions have been specialized and CANNOT be dynamic.
[2024-01-09 20:58:33,852] torch._dynamo.eval_frame: [INFO] ```
[2024-01-09 20:58:33,852] torch._dynamo.eval_frame: [INFO] def specializations(x):
[2024-01-09 20:58:33,852] torch._dynamo.eval_frame: [INFO]     # x:
[2024-01-09 20:58:33,852] torch._dynamo.eval_frame: [INFO]     assert x.size()[0] == 1
[2024-01-09 20:58:33,852] torch._dynamo.eval_frame: [INFO]     assert x.size()[1] == 3
[2024-01-09 20:58:33,852] torch._dynamo.eval_frame: [INFO]     assert x.size()[2] == 4
[2024-01-09 20:58:33,852] torch._dynamo.eval_frame: [INFO]     assert x.size()[3] == 4
[2024-01-09 20:58:33,852] torch._dynamo.eval_frame: [INFO] ```
[2024-01-09 20:58:33,852] torch._dynamo.eval_frame: [INFO] 
Value of eta max is: 0.0001
  0%|                                                                                                                                                  | 0/5 [00:00<?, ?it/s]Epoch 1/5....Training loss = 9.3225
[2024-01-09 20:59:23,958] [4/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py:471
[2024-01-09 20:59:23,958] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
  0%|                                                                                                                                                  | 0/5 [00:23<?, ?it/s]
Traceback (most recent call last):
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/train-minimum.py", line 404, in <module>
    aten_dialect: ExportedProgram = export(m, (enc_input, dec_input, dec_source_mask, dec_target_mask))
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/export/__init__.py", line 191, in export
    return _export(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/export/exported_program.py", line 78, in wrapper
    return fn(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/export/_trace.py", line 588, in _export
    gm_torch_level = _export_to_torch_ir(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/export/_trace.py", line 265, in _export_to_torch_ir
    gm_torch_level, _ = torch._dynamo.export(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 1268, in inner
    result_traced = opt_f(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 417, in _fn
    return fn(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 580, in catch_errors
    return callback(frame, cache_entry, hooks, frame_state)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 384, in _convert_frame_assert
    return _compile(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 643, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 247, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 524, in compile_inner
    out_code = transform_code_object(code, transform)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/bytecode_transformation.py", line 1033, in transform_code_object
    transformations(instructions, code_options)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 151, in _fn
    return fn(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 489, in transform
    tracer.run()
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 2098, in run
    super().run()
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 780, in run
    and self.step()
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 743, in step
    getattr(self, inst.opname)(inst)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 462, in wrapper
    return inner_fn(self, inst)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 1190, in CALL_FUNCTION
    self.call_function(fn, args, {})
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 644, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/variables/functions.py", line 301, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/variables/functions.py", line 255, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/variables/functions.py", line 84, in call_function
    return tx.inline_user_function_return(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 650, in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 2233, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 2348, in inline_call_
    tracer.run()
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 780, in run
    and self.step()
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 743, in step
    getattr(self, inst.opname)(inst)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 462, in wrapper
    return inner_fn(self, inst)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 1190, in CALL_FUNCTION
    self.call_function(fn, args, {})
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 644, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/variables/functions.py", line 301, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/variables/functions.py", line 255, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/variables/functions.py", line 84, in call_function
    return tx.inline_user_function_return(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 650, in inline_user_function_return
    return InliningInstructionTranslator.inline_call(self, fn, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 2233, in inline_call
    return cls.inline_call_(parent, func, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 2348, in inline_call_
    tracer.run()
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 780, in run
    and self.step()
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 743, in step
    getattr(self, inst.opname)(inst)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 462, in wrapper
    return inner_fn(self, inst)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 1243, in CALL_FUNCTION_KW
    self.call_function(fn, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 644, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/variables/user_defined.py", line 398, in call_function
    return super().call_function(tx, args, kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/variables/base.py", line 321, in call_function
    unimplemented(f"call_function {self} {args} {kwargs}")
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/exc.py", line 193, in unimplemented
    raise Unsupported(msg)
torch._dynamo.exc.Unsupported: call_function UserDefinedClassVariable() [TensorVariable()] {'requires_grad': ConstantVariable(bool)}

from user code:
   File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py", line 486, in forward
    enc_embed = self.encoder_embedding.forward(enc_input)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py", line 397, in forward
    x = self.pos_encoding.forward(x)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py", line 353, in forward
    x = x + Variable(self.pe[:, :x.size(1)], requires_grad=False)

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information

[2024-01-09 20:59:23,981] torch._dynamo.utils: [INFO] TorchDynamo compilation metrics:
[2024-01-09 20:59:23,981] torch._dynamo.utils: [INFO] Function                           Runtimes (s)
[2024-01-09 20:59:23,981] torch._dynamo.utils: [INFO] -------------------------------  --------------
[2024-01-09 20:59:23,981] torch._dynamo.utils: [INFO] _compile.<locals>.compile_inner          0.1496
[2024-01-09 20:59:23,981] torch._dynamo.utils: [INFO] OutputGraph.call_user_compiler           0.0004
[2024-01-09 20:59:23,981] torch._dynamo.utils: [INFO] create_aot_dispatcher_function           0.356
(base) adonnini1@actlnxlptp6:~/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master$

@angelayi
Copy link
Contributor

Seems like we've reached a new error message -- It looks like it's pointing to the line:

  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py", line 353, in forward
    x = x + Variable(self.pe[:, :x.size(1)], requires_grad=False)

What is this Variable class? Does it override __new__?

@angelayi
Copy link
Contributor

ok, I found your repo and made a minimum repro:

      from torch.autograd import Variable

      class M(torch.nn.Module):
          def __init__(self):
              super().__init__()
              self.pe = torch.zeros(5000, 1024)

          def forward(self, x):
              x = x + Variable(self.pe[:, : x.size(1)], requires_grad=False)
              return x

      torch.export.export(M(), (torch.ones(1, 5000),))

Why are you using Variable? It looks like it's deprecated: https://pytorch.org/docs/stable/autograd.html#variable-deprecated

But also it seems like you're just using it to set requires_grad=False. Why not just use torch.detach()?

@adonnini
Copy link
Author

@angelayi

After reading this comment, please red the next one which I am about to post

The "Variable" issue could be a red herring. I am aware that Variable is deprecated.

I ran into the same error when lowering the model onto an Android device using pytorch mobile and torchscript. When running the training code, execution failed with this error after running five epochs (the first four epochs were successful). The solution was to comment out the validation loop executed after each training execution within the epoch loop.

Before using the above solution, I tried fixing the "Variable" problem replacing in model.py

x = x + Variable(self.pe[:, :x.size(1)], requires_grad=False)

with

x = x + torch.tensor(self.pe[:, :x.size(1)], requires_grad=False)

this was a "clumsy" solution (please keep in mind that I am a Java developer) but it did the job. However, execution then failed with another error.

The same happened in this case. I used the same solution as described above. The validation loop is sill commented out.

However, execution still failed. Below you will find the latest traceback log.

Please note that the statement with Variable in model.py does not cause any problems when running the model with all executorch code commented out.

Thanks

TRACEBACK LOG

(base) adonnini1@actlnxlptp6:~/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master$ python3 train-minimum.py
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py:186: UserWarning: This function is deprecated. Please use torch.export.export instead.
  warnings.warn("This function is deprecated. Please use torch.export.export instead.")
[2024-01-10 14:58:31,545] [0/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/backends/xnnpack/partition/graphs/bilinear_2d.py:19
[2024-01-10 14:58:31,546] [0/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2024-01-10 14:58:31,638] [0/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2024-01-10 14:58:31,639] [0/0] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function dynamo_normalization_capturing_compiler
[2024-01-10 14:58:31,639] [0/0] torch._dynamo.output_graph: [INFO] Step 2: done compiler function dynamo_normalization_capturing_compiler
[2024-01-10 14:58:31,640] [0/0] torch.fx.experimental.symbolic_shapes: [INFO] produce_guards
[2024-01-10 14:58:31,656] torch._dynamo.eval_frame: [INFO] Summary of dimension constraints:
[2024-01-10 14:58:31,656] torch._dynamo.eval_frame: [INFO] The following dimensions have been specialized and CANNOT be dynamic.
[2024-01-10 14:58:31,656] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 14:58:31,656] torch._dynamo.eval_frame: [INFO] def specializations(x):
[2024-01-10 14:58:31,656] torch._dynamo.eval_frame: [INFO]     # x:
[2024-01-10 14:58:31,656] torch._dynamo.eval_frame: [INFO]     assert x.size()[0] == 1
[2024-01-10 14:58:31,656] torch._dynamo.eval_frame: [INFO]     assert x.size()[1] == 3
[2024-01-10 14:58:31,656] torch._dynamo.eval_frame: [INFO]     assert x.size()[2] == 4
[2024-01-10 14:58:31,656] torch._dynamo.eval_frame: [INFO]     assert x.size()[3] == 4
[2024-01-10 14:58:31,656] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 14:58:31,656] torch._dynamo.eval_frame: [INFO] 
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py:186: UserWarning: This function is deprecated. Please use torch.export.export instead.
  warnings.warn("This function is deprecated. Please use torch.export.export instead.")
[2024-01-10 14:58:31,916] [1/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/backends/xnnpack/partition/graphs/bilinear_2d.py:19
[2024-01-10 14:58:31,916] [1/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2024-01-10 14:58:31,927] [1/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2024-01-10 14:58:31,927] [1/0] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function dynamo_normalization_capturing_compiler
[2024-01-10 14:58:31,927] [1/0] torch._dynamo.output_graph: [INFO] Step 2: done compiler function dynamo_normalization_capturing_compiler
[2024-01-10 14:58:31,928] [1/0] torch.fx.experimental.symbolic_shapes: [INFO] produce_guards
[2024-01-10 14:58:31,942] torch._dynamo.eval_frame: [INFO] Summary of dimension constraints:
[2024-01-10 14:58:31,942] torch._dynamo.eval_frame: [INFO] The following dimensions have been specialized and CANNOT be dynamic.
[2024-01-10 14:58:31,942] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 14:58:31,942] torch._dynamo.eval_frame: [INFO] def specializations(x):
[2024-01-10 14:58:31,942] torch._dynamo.eval_frame: [INFO]     # x:
[2024-01-10 14:58:31,942] torch._dynamo.eval_frame: [INFO]     assert x.size()[0] == 1
[2024-01-10 14:58:31,942] torch._dynamo.eval_frame: [INFO]     assert x.size()[1] == 3
[2024-01-10 14:58:31,942] torch._dynamo.eval_frame: [INFO]     assert x.size()[2] == 4
[2024-01-10 14:58:31,942] torch._dynamo.eval_frame: [INFO]     assert x.size()[3] == 4
[2024-01-10 14:58:31,942] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 14:58:31,942] torch._dynamo.eval_frame: [INFO] 
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py:186: UserWarning: This function is deprecated. Please use torch.export.export instead.
  warnings.warn("This function is deprecated. Please use torch.export.export instead.")
[2024-01-10 14:58:32,194] [2/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/backends/xnnpack/partition/graphs/bilinear_2d.py:19
[2024-01-10 14:58:32,194] [2/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2024-01-10 14:58:32,205] [2/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2024-01-10 14:58:32,205] [2/0] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function dynamo_normalization_capturing_compiler
[2024-01-10 14:58:32,206] [2/0] torch._dynamo.output_graph: [INFO] Step 2: done compiler function dynamo_normalization_capturing_compiler
[2024-01-10 14:58:32,206] [2/0] torch.fx.experimental.symbolic_shapes: [INFO] produce_guards
[2024-01-10 14:58:32,221] torch._dynamo.eval_frame: [INFO] Summary of dimension constraints:
[2024-01-10 14:58:32,221] torch._dynamo.eval_frame: [INFO] The following dimensions have been specialized and CANNOT be dynamic.
[2024-01-10 14:58:32,221] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 14:58:32,221] torch._dynamo.eval_frame: [INFO] def specializations(x):
[2024-01-10 14:58:32,221] torch._dynamo.eval_frame: [INFO]     # x:
[2024-01-10 14:58:32,221] torch._dynamo.eval_frame: [INFO]     assert x.size()[0] == 1
[2024-01-10 14:58:32,221] torch._dynamo.eval_frame: [INFO]     assert x.size()[1] == 3
[2024-01-10 14:58:32,221] torch._dynamo.eval_frame: [INFO]     assert x.size()[2] == 4
[2024-01-10 14:58:32,221] torch._dynamo.eval_frame: [INFO]     assert x.size()[3] == 4
[2024-01-10 14:58:32,221] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 14:58:32,221] torch._dynamo.eval_frame: [INFO] 
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py:186: UserWarning: This function is deprecated. Please use torch.export.export instead.
  warnings.warn("This function is deprecated. Please use torch.export.export instead.")
[2024-01-10 14:58:32,554] [3/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/backends/xnnpack/partition/graphs/bilinear_2d.py:19
[2024-01-10 14:58:32,554] [3/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2024-01-10 14:58:32,565] [3/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2024-01-10 14:58:32,566] [3/0] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function dynamo_normalization_capturing_compiler
[2024-01-10 14:58:32,566] [3/0] torch._dynamo.output_graph: [INFO] Step 2: done compiler function dynamo_normalization_capturing_compiler
[2024-01-10 14:58:32,567] [3/0] torch.fx.experimental.symbolic_shapes: [INFO] produce_guards
[2024-01-10 14:58:32,581] torch._dynamo.eval_frame: [INFO] Summary of dimension constraints:
[2024-01-10 14:58:32,581] torch._dynamo.eval_frame: [INFO] The following dimensions have been specialized and CANNOT be dynamic.
[2024-01-10 14:58:32,581] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 14:58:32,581] torch._dynamo.eval_frame: [INFO] def specializations(x):
[2024-01-10 14:58:32,581] torch._dynamo.eval_frame: [INFO]     # x:
[2024-01-10 14:58:32,581] torch._dynamo.eval_frame: [INFO]     assert x.size()[0] == 1
[2024-01-10 14:58:32,581] torch._dynamo.eval_frame: [INFO]     assert x.size()[1] == 3
[2024-01-10 14:58:32,581] torch._dynamo.eval_frame: [INFO]     assert x.size()[2] == 4
[2024-01-10 14:58:32,581] torch._dynamo.eval_frame: [INFO]     assert x.size()[3] == 4
[2024-01-10 14:58:32,581] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 14:58:32,581] torch._dynamo.eval_frame: [INFO] 
/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py:353: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  x = x + torch.tensor(self.pe[:, :x.size(1)], requires_grad=False)
Value of eta max is: 0.0001
  0%|                                                                                                                                                  | 0/5 [00:00<?, ?it/s]Epoch 1/5....Training loss = 8.7304
[2024-01-10 14:59:22,206] [4/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py:474
[2024-01-10 14:59:22,206] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/utils.py:1666: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  return node.target(*args, **kwargs)
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/utils.py:1666: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  return node.target(*args, **kwargs)
[2024-01-10 14:59:23,038] [4/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2024-01-10 14:59:23,050] [4/0] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function dynamo_normalization_capturing_compiler
[2024-01-10 14:59:23,050] [4/0] torch._dynamo.output_graph: [INFO] Step 2: done compiler function dynamo_normalization_capturing_compiler
[2024-01-10 14:59:23,057] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] produce_guards
<eval_with_key>.201:13: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  tensor = torch.tensor(getitem, requires_grad = False);  getitem = None
<eval_with_key>.201:201: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  tensor_1 = torch.tensor(getitem_1, requires_grad = False);  getitem_1 = None
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO] Summary of dimension constraints:
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO] The following dimensions have been specialized and CANNOT be dynamic.
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO] def specializations(enc_input, dec_input, dec_source_mask, dec_target_mask):
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO]     # enc_input:
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO]     assert enc_input.size()[0] == 27
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO]     assert enc_input.size()[1] == 7
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO]     assert enc_input.size()[2] == 2
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO] 
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO]     # dec_input:
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO]     assert dec_input.size()[0] == 27
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO]     assert dec_input.size()[1] == 12
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO]     assert dec_input.size()[2] == 3
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO] 
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO]     # dec_source_mask:
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO]     assert dec_source_mask.size()[0] == 27
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO]     assert dec_source_mask.size()[1] == 1
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO]     assert dec_source_mask.size()[2] == 7
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO] 
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO]     # dec_target_mask:
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO]     assert dec_target_mask.size()[0] == 27
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO]     assert dec_target_mask.size()[1] == 12
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO]     assert dec_target_mask.size()[2] == 12
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 14:59:23,426] torch._dynamo.eval_frame: [INFO] 
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/export/_safeguard.py:38: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  return func(*args, **kwargs)
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/export/_safeguard.py:38: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  return func(*args, **kwargs)
  0%|                                                                                                                                                  | 0/5 [00:51<?, ?it/s]
Traceback (most recent call last):
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/train-minimum.py", line 416, in <module>
    edge_program = edge_program.to_backend(XnnpackPartitioner)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/exir/program/_program.py", line 999, in to_backend
    new_edge_programs[name] = to_backend(program, partitioner)
  File "/home/adonnini1/anaconda3/lib/python3.9/functools.py", line 888, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/exir/backend/backend_api.py", line 331, in _
    partitioner_result = partitioner_instance(copied_edge_program)
TypeError: __init__() takes 1 positional argument but 2 were given
[2024-01-10 14:59:50,230] torch._dynamo.utils: [INFO] TorchDynamo compilation metrics:
[2024-01-10 14:59:50,230] torch._dynamo.utils: [INFO] Function                           Runtimes (s)
[2024-01-10 14:59:50,230] torch._dynamo.utils: [INFO] -------------------------------  --------------
[2024-01-10 14:59:50,230] torch._dynamo.utils: [INFO] _compile.<locals>.compile_inner          1.0225
[2024-01-10 14:59:50,230] torch._dynamo.utils: [INFO] OutputGraph.call_user_compiler           0.0007
[2024-01-10 14:59:50,230] torch._dynamo.utils: [INFO] create_aot_dispatcher_function           3.9006
(base) adonnini1@actlnxlptp6:~/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master$

@adonnini
Copy link
Author

@angelayi
In addition to the "clumsy" Variable problem fix on model.py, I also noticed that I did not include constraints in

aten_dialect: ExportedProgram = export(m, (enc_input, dec_input, dec_source_mask, dec_target_mask))

I changed it to

 aten_dialect: ExportedProgram = export(m, (enc_input, dec_input, dec_source_mask, dec_target_mask), constraints=constraints)

where

        constraints = [
            dynamic_dim(enc_input_tensor, 0),
            dynamic_dim(dec_input, 0),
            dynamic_dim(dec_source_mask, 0),
            dynamic_dim(dec_target_mask, 0),
        ]

Now, execution fails producing the traceback log reported below.

Thanks

TRACEBACK LOG

(base) adonnini1@actlnxlptp6:~/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master$ python3 train-minimum.py
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py:186: UserWarning: This function is deprecated. Please use torch.export.export instead.
  warnings.warn("This function is deprecated. Please use torch.export.export instead.")
[2024-01-10 15:43:40,806] [0/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/backends/xnnpack/partition/graphs/bilinear_2d.py:19
[2024-01-10 15:43:40,807] [0/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2024-01-10 15:43:40,900] [0/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2024-01-10 15:43:40,900] [0/0] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function dynamo_normalization_capturing_compiler
[2024-01-10 15:43:40,901] [0/0] torch._dynamo.output_graph: [INFO] Step 2: done compiler function dynamo_normalization_capturing_compiler
[2024-01-10 15:43:40,902] [0/0] torch.fx.experimental.symbolic_shapes: [INFO] produce_guards
[2024-01-10 15:43:40,918] torch._dynamo.eval_frame: [INFO] Summary of dimension constraints:
[2024-01-10 15:43:40,918] torch._dynamo.eval_frame: [INFO] The following dimensions have been specialized and CANNOT be dynamic.
[2024-01-10 15:43:40,918] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 15:43:40,918] torch._dynamo.eval_frame: [INFO] def specializations(x):
[2024-01-10 15:43:40,918] torch._dynamo.eval_frame: [INFO]     # x:
[2024-01-10 15:43:40,918] torch._dynamo.eval_frame: [INFO]     assert x.size()[0] == 1
[2024-01-10 15:43:40,918] torch._dynamo.eval_frame: [INFO]     assert x.size()[1] == 3
[2024-01-10 15:43:40,918] torch._dynamo.eval_frame: [INFO]     assert x.size()[2] == 4
[2024-01-10 15:43:40,918] torch._dynamo.eval_frame: [INFO]     assert x.size()[3] == 4
[2024-01-10 15:43:40,918] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 15:43:40,918] torch._dynamo.eval_frame: [INFO] 
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py:186: UserWarning: This function is deprecated. Please use torch.export.export instead.
  warnings.warn("This function is deprecated. Please use torch.export.export instead.")
[2024-01-10 15:43:41,180] [1/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/backends/xnnpack/partition/graphs/bilinear_2d.py:19
[2024-01-10 15:43:41,181] [1/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2024-01-10 15:43:41,191] [1/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2024-01-10 15:43:41,192] [1/0] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function dynamo_normalization_capturing_compiler
[2024-01-10 15:43:41,192] [1/0] torch._dynamo.output_graph: [INFO] Step 2: done compiler function dynamo_normalization_capturing_compiler
[2024-01-10 15:43:41,192] [1/0] torch.fx.experimental.symbolic_shapes: [INFO] produce_guards
[2024-01-10 15:43:41,206] torch._dynamo.eval_frame: [INFO] Summary of dimension constraints:
[2024-01-10 15:43:41,206] torch._dynamo.eval_frame: [INFO] The following dimensions have been specialized and CANNOT be dynamic.
[2024-01-10 15:43:41,206] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 15:43:41,206] torch._dynamo.eval_frame: [INFO] def specializations(x):
[2024-01-10 15:43:41,206] torch._dynamo.eval_frame: [INFO]     # x:
[2024-01-10 15:43:41,206] torch._dynamo.eval_frame: [INFO]     assert x.size()[0] == 1
[2024-01-10 15:43:41,206] torch._dynamo.eval_frame: [INFO]     assert x.size()[1] == 3
[2024-01-10 15:43:41,206] torch._dynamo.eval_frame: [INFO]     assert x.size()[2] == 4
[2024-01-10 15:43:41,206] torch._dynamo.eval_frame: [INFO]     assert x.size()[3] == 4
[2024-01-10 15:43:41,206] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 15:43:41,206] torch._dynamo.eval_frame: [INFO] 
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py:186: UserWarning: This function is deprecated. Please use torch.export.export instead.
  warnings.warn("This function is deprecated. Please use torch.export.export instead.")
[2024-01-10 15:43:41,461] [2/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/backends/xnnpack/partition/graphs/bilinear_2d.py:19
[2024-01-10 15:43:41,461] [2/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2024-01-10 15:43:41,472] [2/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2024-01-10 15:43:41,473] [2/0] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function dynamo_normalization_capturing_compiler
[2024-01-10 15:43:41,473] [2/0] torch._dynamo.output_graph: [INFO] Step 2: done compiler function dynamo_normalization_capturing_compiler
[2024-01-10 15:43:41,473] [2/0] torch.fx.experimental.symbolic_shapes: [INFO] produce_guards
[2024-01-10 15:43:41,488] torch._dynamo.eval_frame: [INFO] Summary of dimension constraints:
[2024-01-10 15:43:41,488] torch._dynamo.eval_frame: [INFO] The following dimensions have been specialized and CANNOT be dynamic.
[2024-01-10 15:43:41,488] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 15:43:41,488] torch._dynamo.eval_frame: [INFO] def specializations(x):
[2024-01-10 15:43:41,488] torch._dynamo.eval_frame: [INFO]     # x:
[2024-01-10 15:43:41,488] torch._dynamo.eval_frame: [INFO]     assert x.size()[0] == 1
[2024-01-10 15:43:41,488] torch._dynamo.eval_frame: [INFO]     assert x.size()[1] == 3
[2024-01-10 15:43:41,488] torch._dynamo.eval_frame: [INFO]     assert x.size()[2] == 4
[2024-01-10 15:43:41,488] torch._dynamo.eval_frame: [INFO]     assert x.size()[3] == 4
[2024-01-10 15:43:41,488] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 15:43:41,488] torch._dynamo.eval_frame: [INFO] 
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py:186: UserWarning: This function is deprecated. Please use torch.export.export instead.
  warnings.warn("This function is deprecated. Please use torch.export.export instead.")
[2024-01-10 15:43:41,826] [3/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/backends/xnnpack/partition/graphs/bilinear_2d.py:19
[2024-01-10 15:43:41,826] [3/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2024-01-10 15:43:41,837] [3/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2024-01-10 15:43:41,838] [3/0] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function dynamo_normalization_capturing_compiler
[2024-01-10 15:43:41,838] [3/0] torch._dynamo.output_graph: [INFO] Step 2: done compiler function dynamo_normalization_capturing_compiler
[2024-01-10 15:43:41,839] [3/0] torch.fx.experimental.symbolic_shapes: [INFO] produce_guards
[2024-01-10 15:43:41,853] torch._dynamo.eval_frame: [INFO] Summary of dimension constraints:
[2024-01-10 15:43:41,853] torch._dynamo.eval_frame: [INFO] The following dimensions have been specialized and CANNOT be dynamic.
[2024-01-10 15:43:41,853] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 15:43:41,853] torch._dynamo.eval_frame: [INFO] def specializations(x):
[2024-01-10 15:43:41,853] torch._dynamo.eval_frame: [INFO]     # x:
[2024-01-10 15:43:41,853] torch._dynamo.eval_frame: [INFO]     assert x.size()[0] == 1
[2024-01-10 15:43:41,853] torch._dynamo.eval_frame: [INFO]     assert x.size()[1] == 3
[2024-01-10 15:43:41,853] torch._dynamo.eval_frame: [INFO]     assert x.size()[2] == 4
[2024-01-10 15:43:41,853] torch._dynamo.eval_frame: [INFO]     assert x.size()[3] == 4
[2024-01-10 15:43:41,853] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 15:43:41,853] torch._dynamo.eval_frame: [INFO] 
/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py:353: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  x = x + torch.tensor(self.pe[:, :x.size(1)], requires_grad=False)
Value of eta max is: 0.0000
  0%|                                                                                                                                                  | 0/5 [00:00<?, ?it/s]Epoch 1/5....Training loss = 7.1915
/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/train-minimum.py:404: DeprecationWarning: Using `constraints` to specify dynamic shapes for export is DEPRECATED and will not be supported in the future. Please use `dynamic_shapes` instead (see docs on `torch.export.export`).
  aten_dialect: ExportedProgram = export(m, (enc_input, dec_input, dec_source_mask, dec_target_mask), constraints=constraints)
[2024-01-10 15:44:31,664] [4/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py:474
[2024-01-10 15:44:31,664] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2024-01-10 15:44:31,691] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] create_symbol s0 = 27 for L['enc_input'].size()[0] [2, 9223372036854775806]
[2024-01-10 15:44:31,695] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] create_symbol s1 = 27 for L['dec_input'].size()[0] [2, 9223372036854775806]
[2024-01-10 15:44:31,697] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] create_symbol s2 = 27 for L['dec_source_mask'].size()[0] [2, 9223372036854775806]
[2024-01-10 15:44:31,700] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] create_symbol s3 = 27 for L['dec_target_mask'].size()[0] [2, 9223372036854775806]
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/utils.py:1666: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  return node.target(*args, **kwargs)
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/utils.py:1666: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  return node.target(*args, **kwargs)
[2024-01-10 15:44:32,290] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] set_replacement s3 = s1
[2024-01-10 15:44:32,290] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] set_replacement s3 = s1
[2024-01-10 15:44:32,291] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] eval Eq(s1, s3) [guard added] at Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/utils.py:53 in attention (_refs/__init__.py:401 in _broadcast_shapes)
[2024-01-10 15:44:32,352] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] eval Eq(Mod(3584*s0, 512*s1), 0) [guard added] at Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py:77 in forward (_prims_common/__init__.py:871 in infer_size)
[2024-01-10 15:44:32,449] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] set_replacement s1 = s0
[2024-01-10 15:44:32,449] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] set_replacement s1 = s0
[2024-01-10 15:44:32,449] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] eval Eq(s0, s1) [guard added] at Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py:77 in forward (_refs/__init__.py:3561 in _reshape_view_helper)
[2024-01-10 15:44:32,466] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] set_replacement s2 = s0
[2024-01-10 15:44:32,467] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] set_replacement s2 = s0
[2024-01-10 15:44:32,467] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] eval Eq(s0, s2) [guard added] at Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/utils.py:53 in attention (_refs/__init__.py:401 in _broadcast_shapes)
[2024-01-10 15:44:32,467] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] set_replacement s1 = s0
[2024-01-10 15:44:32,527] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] set_replacement s3 = s0
[2024-01-10 15:44:33,217] [4/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2024-01-10 15:44:33,232] [4/0] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function dynamo_normalization_capturing_compiler
[2024-01-10 15:44:33,232] [4/0] torch._dynamo.output_graph: [INFO] Step 2: done compiler function dynamo_normalization_capturing_compiler
[2024-01-10 15:44:33,239] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] produce_guards
[2024-01-10 15:44:33,241] [4/0] torch._guards: [ERROR] Error while creating guard:
[2024-01-10 15:44:33,241] [4/0] torch._guards: [ERROR] Name: ''
[2024-01-10 15:44:33,241] [4/0] torch._guards: [ERROR]     Source: shape_env
[2024-01-10 15:44:33,241] [4/0] torch._guards: [ERROR]     Create Function: SHAPE_ENV
[2024-01-10 15:44:33,241] [4/0] torch._guards: [ERROR]     Guard Types: None
[2024-01-10 15:44:33,241] [4/0] torch._guards: [ERROR]     Code List: None
[2024-01-10 15:44:33,241] [4/0] torch._guards: [ERROR]     Object Weakref: None
[2024-01-10 15:44:33,241] [4/0] torch._guards: [ERROR]     Guarded Class Weakref: None
[2024-01-10 15:44:33,241] [4/0] torch._guards: [ERROR] Created at:
[2024-01-10 15:44:33,241] [4/0] torch._guards: [ERROR]   File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 471, in transform
[2024-01-10 15:44:33,241] [4/0] torch._guards: [ERROR]     tracer = InstructionTranslator(
[2024-01-10 15:44:33,241] [4/0] torch._guards: [ERROR]   File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/symbolic_convert.py", line 2029, in __init__
[2024-01-10 15:44:33,241] [4/0] torch._guards: [ERROR]     output=OutputGraph(
[2024-01-10 15:44:33,241] [4/0] torch._guards: [ERROR]   File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/output_graph.py", line 297, in __init__
[2024-01-10 15:44:33,241] [4/0] torch._guards: [ERROR]     self.init_ambient_guards()
[2024-01-10 15:44:33,241] [4/0] torch._guards: [ERROR]   File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/output_graph.py", line 375, in init_ambient_guards
[2024-01-10 15:44:33,241] [4/0] torch._guards: [ERROR]     self.guards.add(ShapeEnvSource().make_guard(GuardBuilder.SHAPE_ENV))
  0%|                                                                                                                                                  | 0/5 [00:24<?, ?it/s]
Traceback (most recent call last):
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/export/_trace.py", line 265, in _export_to_torch_ir
    gm_torch_level, _ = torch._dynamo.export(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 1308, in inner
    raise constraint_violation_error
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 1268, in inner
    result_traced = opt_f(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 417, in _fn
    return fn(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/eval_frame.py", line 580, in catch_errors
    return callback(frame, cache_entry, hooks, frame_state)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 384, in _convert_frame_assert
    return _compile(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 643, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/utils.py", line 247, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 623, in compile_inner
    check_fn = CheckFunctionManager(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/guards.py", line 993, in __init__
    guard.create(builder)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_guards.py", line 246, in create
    return self.create_fn(builder, self)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/guards.py", line 645, in SHAPE_ENV
    guards = output_graph.shape_env.produce_guards(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/fx/experimental/symbolic_shapes.py", line 3019, in produce_guards
    raise ConstraintViolationError(
torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['dec_input'].size()[0], L['dec_source_mask'].size()[0], L['dec_target_mask'].size()[0])! For more information, run with TORCH_LOGS=dynamic.
  - The values of L['dec_input'].size()[0] = L['dec_input'].size()[0] and L['enc_input'].size()[0] = L['enc_input'].size()[0] must always be equal.
  - The values of L['dec_source_mask'].size()[0] = L['dec_source_mask'].size()[0] and L['enc_input'].size()[0] = L['enc_input'].size()[0] must always be equal.
  - The values of L['dec_target_mask'].size()[0] = L['dec_target_mask'].size()[0] and L['enc_input'].size()[0] = L['enc_input'].size()[0] must always be equal.

The following dimensions have been specialized and CANNOT be dynamic.

def specializations(enc_input, dec_input, dec_source_mask, dec_target_mask):
# enc_input:
assert enc_input.size()[1] == 7
assert enc_input.size()[2] == 2

# dec_input:
assert dec_input.size()[1] == 12
assert dec_input.size()[2] == 3

# dec_source_mask:
assert dec_source_mask.size()[1] == 1
assert dec_source_mask.size()[2] == 7

# dec_target_mask:
assert dec_target_mask.size()[1] == 12
assert dec_target_mask.size()[2] == 12


The following dimensions CAN be dynamic.
Please use the following code to specify the constraints they must satisfy:

def specify_constraints(enc_input, dec_input, dec_source_mask, dec_target_mask):
return [
# dec_input:
dynamic_dim(dec_input, 0) == dynamic_dim(enc_input, 0),

    # dec_source_mask:
    dynamic_dim(dec_source_mask, 0) == dynamic_dim(enc_input, 0),

    # dec_target_mask:
    dynamic_dim(dec_target_mask, 0) == dynamic_dim(enc_input, 0),
]



During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/train-minimum.py", line 404, in <module>
    aten_dialect: ExportedProgram = export(m, (enc_input, dec_input, dec_source_mask, dec_target_mask), constraints=constraints)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/export/__init__.py", line 191, in export
    return _export(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/export/exported_program.py", line 78, in wrapper
    return fn(*args, **kwargs)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/export/_trace.py", line 588, in _export
    gm_torch_level = _export_to_torch_ir(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/export/_trace.py", line 276, in _export_to_torch_ir
    raise UserError(UserErrorType.CONSTRAINT_VIOLATION, str(e))  # noqa: TRY200
torch._dynamo.exc.UserError: Constraints violated (L['dec_input'].size()[0], L['dec_source_mask'].size()[0], L['dec_target_mask'].size()[0])! For more information, run with TORCH_LOGS=dynamic.
  - The values of L['dec_input'].size()[0] = L['dec_input'].size()[0] and L['enc_input'].size()[0] = L['enc_input'].size()[0] must always be equal.
  - The values of L['dec_source_mask'].size()[0] = L['dec_source_mask'].size()[0] and L['enc_input'].size()[0] = L['enc_input'].size()[0] must always be equal.
  - The values of L['dec_target_mask'].size()[0] = L['dec_target_mask'].size()[0] and L['enc_input'].size()[0] = L['enc_input'].size()[0] must always be equal.

The following dimensions have been specialized and CANNOT be dynamic.

def specializations(enc_input, dec_input, dec_source_mask, dec_target_mask):
# enc_input:
assert enc_input.size()[1] == 7
assert enc_input.size()[2] == 2

# dec_input:
assert dec_input.size()[1] == 12
assert dec_input.size()[2] == 3

# dec_source_mask:
assert dec_source_mask.size()[1] == 1
assert dec_source_mask.size()[2] == 7

# dec_target_mask:
assert dec_target_mask.size()[1] == 12
assert dec_target_mask.size()[2] == 12


The following dimensions CAN be dynamic.
Please use the following code to specify the constraints they must satisfy:

def specify_constraints(enc_input, dec_input, dec_source_mask, dec_target_mask):
return [
# dec_input:
dynamic_dim(dec_input, 0) == dynamic_dim(enc_input, 0),

    # dec_source_mask:
    dynamic_dim(dec_source_mask, 0) == dynamic_dim(enc_input, 0),

    # dec_target_mask:
    dynamic_dim(dec_target_mask, 0) == dynamic_dim(enc_input, 0),
]


[2024-01-10 15:44:33,248] torch._dynamo.utils: [INFO] TorchDynamo compilation metrics:
[2024-01-10 15:44:33,248] torch._dynamo.utils: [INFO] Function                           Runtimes (s)
[2024-01-10 15:44:33,248] torch._dynamo.utils: [INFO] -------------------------------  --------------
[2024-01-10 15:44:33,248] torch._dynamo.utils: [INFO] _compile.<locals>.compile_inner          0.1548
[2024-01-10 15:44:33,248] torch._dynamo.utils: [INFO] OutputGraph.call_user_compiler           0.0007
[2024-01-10 15:44:33,248] torch._dynamo.utils: [INFO] create_aot_dispatcher_function           0.3646
(base) adonnini1@actlnxlptp6:~/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master$

@angelayi
Copy link
Contributor

Please note that the statement with Variable in model.py does not cause any problems when running the model with all executorch code commented out.

Makes sense, I think any old code with Variable should work and continue to work, it's not guaranteed that it will work with new features. It's possible to add support for this, but since it's deprecated already I'm leaning towards not adding support 😅

When running the training code, execution failed with this error after running five epochs (the first four epochs were successful)

Just to clarify, ExecuTorch does not have training support right now. It's only for inference.

TypeError: init() takes 1 positional argument but 2 were given

Regarding the first error, it looks like your code first calls edge_program.to_executorch(), saves it, and then call edge_program.to_backend(). Can you delete the to_executorch() part and try? I wonder if something there is affecting the call to to_backend.

torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['dec_input'].size()[0], L['dec_source_mask'].size()[0], L['dec_target_mask'].size()[0])

Regarding the second error, the error message tells you how to fix it. While tracing, it was found that dec_input, dec_source_mask, and dec_target_mask were found to be equal to each other. So you need to add the following additional constraints:

dynamic_dim(dec_input, 0) == dynamic_dim(enc_input, 0),
dynamic_dim(dec_source_mask, 0) == dynamic_dim(enc_input, 0),
dynamic_dim(dec_target_mask, 0) == dynamic_dim(enc_input, 0),

@adonnini
Copy link
Author

adonnini commented Jan 10, 2024 •
edited
Loading

@angelayi Thanks!
I do understand that executorch does not include training support. I was just referring to the module in my application where I added executorch code.

I made the changes you suggested. Below you will find the code changes I made and the traceback log produced after the changes.

I am sorry about this. Thanks for all your help in helping me resolve this problem.

CODE CHANGES

def specify_constraints(enc_input, dec_input, dec_source_mask, dec_target_mask):
    return [
        # dec_input:
        dynamic_dim(dec_input, 0) == dynamic_dim(enc_input, 0),

        # dec_source_mask:
        dynamic_dim(dec_source_mask, 0) == dynamic_dim(enc_input, 0),

        # dec_target_mask:
        dynamic_dim(dec_target_mask, 0) == dynamic_dim(enc_input, 0),
    ]

        aten_dialect: ExportedProgram = export(m, (enc_input, dec_input, dec_source_mask, dec_target_mask),
                                               constraints=specify_constraints(enc_input, dec_input, dec_source_mask,
                                                                               dec_target_mask))

        # executorch_program: exir.ExecutorchProgramManager = edge_program.to_executorch(
        # ExecutorchBackendConfig(
        # # passes=[],  # User-defined passes
        # )
        # )
        #
        # with open("/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/models/tfmodel.pte", "wb") as file:
        #     file.write(executorch_program.buffer)

TRACEBACK LOG

(base) adonnini1@actlnxlptp6:~/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master$ python3 train-minimum.py
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py:186: UserWarning: This function is deprecated. Please use torch.export.export instead.
  warnings.warn("This function is deprecated. Please use torch.export.export instead.")
[2024-01-10 17:12:44,750] [0/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/backends/xnnpack/partition/graphs/bilinear_2d.py:19
[2024-01-10 17:12:44,750] [0/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2024-01-10 17:12:44,841] [0/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2024-01-10 17:12:44,841] [0/0] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function dynamo_normalization_capturing_compiler
[2024-01-10 17:12:44,841] [0/0] torch._dynamo.output_graph: [INFO] Step 2: done compiler function dynamo_normalization_capturing_compiler
[2024-01-10 17:12:44,842] [0/0] torch.fx.experimental.symbolic_shapes: [INFO] produce_guards
[2024-01-10 17:12:44,858] torch._dynamo.eval_frame: [INFO] Summary of dimension constraints:
[2024-01-10 17:12:44,858] torch._dynamo.eval_frame: [INFO] The following dimensions have been specialized and CANNOT be dynamic.
[2024-01-10 17:12:44,858] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 17:12:44,858] torch._dynamo.eval_frame: [INFO] def specializations(x):
[2024-01-10 17:12:44,858] torch._dynamo.eval_frame: [INFO]     # x:
[2024-01-10 17:12:44,858] torch._dynamo.eval_frame: [INFO]     assert x.size()[0] == 1
[2024-01-10 17:12:44,858] torch._dynamo.eval_frame: [INFO]     assert x.size()[1] == 3
[2024-01-10 17:12:44,858] torch._dynamo.eval_frame: [INFO]     assert x.size()[2] == 4
[2024-01-10 17:12:44,858] torch._dynamo.eval_frame: [INFO]     assert x.size()[3] == 4
[2024-01-10 17:12:44,858] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 17:12:44,858] torch._dynamo.eval_frame: [INFO] 
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py:186: UserWarning: This function is deprecated. Please use torch.export.export instead.
  warnings.warn("This function is deprecated. Please use torch.export.export instead.")
[2024-01-10 17:12:45,118] [1/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/backends/xnnpack/partition/graphs/bilinear_2d.py:19
[2024-01-10 17:12:45,118] [1/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2024-01-10 17:12:45,129] [1/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2024-01-10 17:12:45,130] [1/0] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function dynamo_normalization_capturing_compiler
[2024-01-10 17:12:45,130] [1/0] torch._dynamo.output_graph: [INFO] Step 2: done compiler function dynamo_normalization_capturing_compiler
[2024-01-10 17:12:45,130] [1/0] torch.fx.experimental.symbolic_shapes: [INFO] produce_guards
[2024-01-10 17:12:45,144] torch._dynamo.eval_frame: [INFO] Summary of dimension constraints:
[2024-01-10 17:12:45,144] torch._dynamo.eval_frame: [INFO] The following dimensions have been specialized and CANNOT be dynamic.
[2024-01-10 17:12:45,144] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 17:12:45,144] torch._dynamo.eval_frame: [INFO] def specializations(x):
[2024-01-10 17:12:45,144] torch._dynamo.eval_frame: [INFO]     # x:
[2024-01-10 17:12:45,144] torch._dynamo.eval_frame: [INFO]     assert x.size()[0] == 1
[2024-01-10 17:12:45,144] torch._dynamo.eval_frame: [INFO]     assert x.size()[1] == 3
[2024-01-10 17:12:45,144] torch._dynamo.eval_frame: [INFO]     assert x.size()[2] == 4
[2024-01-10 17:12:45,144] torch._dynamo.eval_frame: [INFO]     assert x.size()[3] == 4
[2024-01-10 17:12:45,144] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 17:12:45,144] torch._dynamo.eval_frame: [INFO] 
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py:186: UserWarning: This function is deprecated. Please use torch.export.export instead.
  warnings.warn("This function is deprecated. Please use torch.export.export instead.")
[2024-01-10 17:12:45,396] [2/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/backends/xnnpack/partition/graphs/bilinear_2d.py:19
[2024-01-10 17:12:45,396] [2/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2024-01-10 17:12:45,407] [2/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2024-01-10 17:12:45,408] [2/0] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function dynamo_normalization_capturing_compiler
[2024-01-10 17:12:45,408] [2/0] torch._dynamo.output_graph: [INFO] Step 2: done compiler function dynamo_normalization_capturing_compiler
[2024-01-10 17:12:45,408] [2/0] torch.fx.experimental.symbolic_shapes: [INFO] produce_guards
[2024-01-10 17:12:45,423] torch._dynamo.eval_frame: [INFO] Summary of dimension constraints:
[2024-01-10 17:12:45,423] torch._dynamo.eval_frame: [INFO] The following dimensions have been specialized and CANNOT be dynamic.
[2024-01-10 17:12:45,423] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 17:12:45,423] torch._dynamo.eval_frame: [INFO] def specializations(x):
[2024-01-10 17:12:45,423] torch._dynamo.eval_frame: [INFO]     # x:
[2024-01-10 17:12:45,423] torch._dynamo.eval_frame: [INFO]     assert x.size()[0] == 1
[2024-01-10 17:12:45,423] torch._dynamo.eval_frame: [INFO]     assert x.size()[1] == 3
[2024-01-10 17:12:45,423] torch._dynamo.eval_frame: [INFO]     assert x.size()[2] == 4
[2024-01-10 17:12:45,423] torch._dynamo.eval_frame: [INFO]     assert x.size()[3] == 4
[2024-01-10 17:12:45,423] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 17:12:45,423] torch._dynamo.eval_frame: [INFO] 
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/__init__.py:186: UserWarning: This function is deprecated. Please use torch.export.export instead.
  warnings.warn("This function is deprecated. Please use torch.export.export instead.")
[2024-01-10 17:12:45,757] [3/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/backends/xnnpack/partition/graphs/bilinear_2d.py:19
[2024-01-10 17:12:45,757] [3/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2024-01-10 17:12:45,769] [3/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2024-01-10 17:12:45,769] [3/0] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function dynamo_normalization_capturing_compiler
[2024-01-10 17:12:45,769] [3/0] torch._dynamo.output_graph: [INFO] Step 2: done compiler function dynamo_normalization_capturing_compiler
[2024-01-10 17:12:45,770] [3/0] torch.fx.experimental.symbolic_shapes: [INFO] produce_guards
[2024-01-10 17:12:45,784] torch._dynamo.eval_frame: [INFO] Summary of dimension constraints:
[2024-01-10 17:12:45,784] torch._dynamo.eval_frame: [INFO] The following dimensions have been specialized and CANNOT be dynamic.
[2024-01-10 17:12:45,784] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 17:12:45,784] torch._dynamo.eval_frame: [INFO] def specializations(x):
[2024-01-10 17:12:45,784] torch._dynamo.eval_frame: [INFO]     # x:
[2024-01-10 17:12:45,784] torch._dynamo.eval_frame: [INFO]     assert x.size()[0] == 1
[2024-01-10 17:12:45,784] torch._dynamo.eval_frame: [INFO]     assert x.size()[1] == 3
[2024-01-10 17:12:45,784] torch._dynamo.eval_frame: [INFO]     assert x.size()[2] == 4
[2024-01-10 17:12:45,784] torch._dynamo.eval_frame: [INFO]     assert x.size()[3] == 4
[2024-01-10 17:12:45,784] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 17:12:45,784] torch._dynamo.eval_frame: [INFO] 
/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py:353: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  x = x + torch.tensor(self.pe[:, :x.size(1)], requires_grad=False)
Value of eta max is: 0.0001
  0%|                                                                                                                                                  | 0/5 [00:00<?, ?it/s]Epoch 1/5....Training loss = 8.5636
/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/train-minimum.py:421: DeprecationWarning: Using `constraints` to specify dynamic shapes for export is DEPRECATED and will not be supported in the future. Please use `dynamic_shapes` instead (see docs on `torch.export.export`).
  aten_dialect: ExportedProgram = export(m, (enc_input, dec_input, dec_source_mask, dec_target_mask),
[2024-01-10 17:13:35,760] [4/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward /home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py:474
[2024-01-10 17:13:35,761] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] create_env
[2024-01-10 17:13:35,785] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] create_symbol s0 = 27 for L['enc_input'].size()[0] [2, 9223372036854775806]
[2024-01-10 17:13:35,789] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] create_symbol s1 = 27 for L['dec_input'].size()[0] [2, 9223372036854775806]
[2024-01-10 17:13:35,792] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] create_symbol s2 = 27 for L['dec_source_mask'].size()[0] [2, 9223372036854775806]
[2024-01-10 17:13:35,794] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] create_symbol s3 = 27 for L['dec_target_mask'].size()[0] [2, 9223372036854775806]
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/utils.py:1666: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  return node.target(*args, **kwargs)
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_dynamo/utils.py:1666: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  return node.target(*args, **kwargs)
[2024-01-10 17:13:36,374] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] set_replacement s3 = s1
[2024-01-10 17:13:36,374] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] set_replacement s3 = s1
[2024-01-10 17:13:36,375] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] eval Eq(s1, s3) [guard added] at Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/utils.py:53 in attention (_refs/__init__.py:401 in _broadcast_shapes)
[2024-01-10 17:13:36,436] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] eval Eq(Mod(3584*s0, 512*s1), 0) [guard added] at Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py:77 in forward (_prims_common/__init__.py:871 in infer_size)
[2024-01-10 17:13:36,530] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] set_replacement s1 = s0
[2024-01-10 17:13:36,530] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] set_replacement s1 = s0
[2024-01-10 17:13:36,530] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] eval Eq(s0, s1) [guard added] at Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/model.py:77 in forward (_refs/__init__.py:3561 in _reshape_view_helper)
[2024-01-10 17:13:36,548] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] set_replacement s2 = s0
[2024-01-10 17:13:36,548] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] set_replacement s2 = s0
[2024-01-10 17:13:36,548] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] eval Eq(s0, s2) [guard added] at Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/utils.py:53 in attention (_refs/__init__.py:401 in _broadcast_shapes)
[2024-01-10 17:13:36,548] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] set_replacement s1 = s0
[2024-01-10 17:13:36,605] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] set_replacement s3 = s0
[2024-01-10 17:13:37,284] [4/0] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2024-01-10 17:13:37,299] [4/0] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function dynamo_normalization_capturing_compiler
[2024-01-10 17:13:37,299] [4/0] torch._dynamo.output_graph: [INFO] Step 2: done compiler function dynamo_normalization_capturing_compiler
[2024-01-10 17:13:37,305] [4/0] torch.fx.experimental.symbolic_shapes: [INFO] produce_guards
<eval_with_key>.201:13: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  tensor = torch.tensor(getitem, requires_grad = False);  getitem = None
<eval_with_key>.201:231: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  tensor_1 = torch.tensor(getitem_43, requires_grad = False);  getitem_43 = None
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO] Summary of dimension constraints:
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO] The following dimensions have been specialized and CANNOT be dynamic.
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO] def specializations(enc_input, dec_input, dec_source_mask, dec_target_mask):
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO]     # enc_input:
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO]     assert enc_input.size()[1] == 7
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO]     assert enc_input.size()[2] == 2
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO] 
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO]     # dec_input:
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO]     assert dec_input.size()[1] == 12
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO]     assert dec_input.size()[2] == 3
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO] 
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO]     # dec_source_mask:
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO]     assert dec_source_mask.size()[1] == 1
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO]     assert dec_source_mask.size()[2] == 7
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO] 
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO]     # dec_target_mask:
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO]     assert dec_target_mask.size()[1] == 12
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO]     assert dec_target_mask.size()[2] == 12
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO] 
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO] The following dimensions CAN be dynamic.
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO] Please use the following code to specify the constraints they must satisfy:
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO] def specify_constraints(enc_input, dec_input, dec_source_mask, dec_target_mask):
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO]     return [
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO]         # dec_input:
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO]         dynamic_dim(dec_input, 0) == dynamic_dim(enc_input, 0),
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO] 
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO]         # dec_source_mask:
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO]         dynamic_dim(dec_source_mask, 0) == dynamic_dim(enc_input, 0),
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO] 
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO]         # dec_target_mask:
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO]         dynamic_dim(dec_target_mask, 0) == dynamic_dim(enc_input, 0),
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO]     ]
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO] ```
[2024-01-10 17:13:38,135] torch._dynamo.eval_frame: [INFO] 
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/export/_safeguard.py:38: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  return func(*args, **kwargs)
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/export/_safeguard.py:38: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  return func(*args, **kwargs)
/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/export/_safeguard.py:38: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  return func(*args, **kwargs)
  0%|                                                                                                                                                  | 0/5 [00:44<?, ?it/s]
Traceback (most recent call last):
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/train-minimum.py", line 426, in <module>
    edge_program: exir.EdgeProgramManager = exir.to_edge(aten_dialect)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/exir/program/_program.py", line 840, in to_edge
    edge_program = ExportedProgram(
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/export/exported_program.py", line 144, in __init__
    self.verifier().check(self)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/verifier.py", line 132, in check
    self._check_graph_module(ep.graph_module)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/verifier.py", line 197, in _check_graph_module
    _check_valid_op(node.target)
  File "/home/adonnini1/anaconda3/lib/python3.9/site-packages/torch/_export/verifier.py", line 180, in _check_valid_op
    self.check_valid_op(op)
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/executorch/exir/verification/verifier.py", line 181, in check_valid_edge_op
    assert op.__name__ in ("alloc",)
AssertionError
[2024-01-10 17:13:57,253] torch._dynamo.utils: [INFO] TorchDynamo compilation metrics:
[2024-01-10 17:13:57,253] torch._dynamo.utils: [INFO] Function                           Runtimes (s)
[2024-01-10 17:13:57,253] torch._dynamo.utils: [INFO] -------------------------------  --------------
[2024-01-10 17:13:57,253] torch._dynamo.utils: [INFO] _compile.<locals>.compile_inner          1.7172
[2024-01-10 17:13:57,253] torch._dynamo.utils: [INFO] OutputGraph.call_user_compiler           0.0007
[2024-01-10 17:13:57,253] torch._dynamo.utils: [INFO] create_aot_dispatcher_function           8.3528
(base) adonnini1@actlnxlptp6:~/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master$

@adonnini
Copy link
Author

@angelayi

I tried an experiment. I went back to first principles, so to speak, and changed the executorch code in my training module to follow exactly what is contained in
https://pytorch.org/executorch/stable/tutorials/export-to-executorch-tutorial.html#lowering-the-whole-module
You will find the code below.
The result was interesting:

  1. The epoch loop (5 epochs) ran to completion

  2. Execution failed with the error reported below. I hope you can tell me how to resolve the issue reported in the traceback log.

I wonder what you make of this.

Thanks

CODE

#FROM https://pytorch.org/executorch/stable/tutorials/export-to-executorch-tutorial.html#lowering-the-whole-module - START
#================================================================================
#Delegating to a Backend - Lowering the Whole Module
#---------------------------------------------------
# Export and lower the module to Edge Dialect
pre_autograd_aten_dialect = capture_pre_autograd_graph(m, (enc_input, dec_input, dec_source_mask, dec_target_mask))
aten_dialect: ExportedProgram = export(pre_autograd_aten_dialect, (enc_input, dec_input, dec_source_mask, dec_target_mask))
# pre_autograd_aten_dialect = capture_pre_autograd_graph(m, (enc_input, dec_input, dec_source_mask, dec_target_mask), constraints=constraints)
# aten_dialect: ExportedProgram = export(pre_autograd_aten_dialect, (enc_input, dec_input, dec_source_mask, dec_target_mask), constraints=constraints)
edge_program: EdgeProgramManager = to_edge(aten_dialect)
to_be_lowered_module = edge_program.exported_program()

from executorch.exir.backend.backend_api import LoweredBackendModule, to_backend

# Lower the module
# lowered_module = to_be_lowered_module.to_backend(XnnpackPartitioner)
lowered_module: LoweredBackendModule = to_backend(
    XnnpackPartitioner, to_be_lowered_module, []
)

print(" - train_minmum - Lowering the Whole Module - pre_autograd_aten_dialect - ", pre_autograd_aten_dialect)
print(" - train_minmum - Lowering the Whole Module - aten_dialect - ", aten_dialect)
print(" - train_minmum - Lowering the Whole Module - edge_program - ", edge_program)
print(" - train_minmum - Lowering the Whole Module - to_be_lowered_module - ", to_be_lowered_module)
print(" - train_minmum - Lowering the Whole Module - lowered_module.backend_id - ", lowered_module.backend_id)
print(" - train_minmum - Lowering the Whole Module - lowered_module.processed_bytes - ", lowered_module.processed_bytes)
print(" - train_minmum - Lowering the Whole Module - lowered_module.original_module - ", lowered_module.original_module)

# Serialize and save it to a file
save_path = save_path = "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/models/TptDelegate.pte"
with open(save_path, "wb") as f:
    f.write(lowered_module.buffer())
#================================================================================
#FROM https://pytorch.org/executorch/stable/tutorials/export-to-executorch-tutorial.html#lowering-the-whole-module - END

TRACEBACK LOG

Traceback (most recent call last):
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/train-minimum.py", line 442, in <module>
    lowered_module: LoweredBackendModule = to_backend(
  File "/home/adonnini1/anaconda3/lib/python3.9/functools.py", line 888, in wrapper
    return dispatch(args[0].__class__)(*args, **kw)
TypeError: to_backend() takes 1 positional argument but 3 were given

@kimishpatel
Copy link
Contributor

@adonnini I looked at your last response for like 30 seconds so ignore if it doenst make sense. to_backend API now takes instance of the class and not just class (doc will have to be updated) . So no XnnpackPartitioner but XnnpackPartitioner().

@adonnini
Copy link
Author

@kimishpatel Thanks! I made the change
from

lowered_module: LoweredBackendModule = to_backend(
    XnnpackPartitioner, to_be_lowered_module, []
)

to

lowered_module: LoweredBackendModule = to_backend(
    XnnpackPartitioner(), to_be_lowered_module, []
)

The same error occurred. I seem to remember that to_backend() changed but I am not sure

@angelayi
Copy link
Contributor

@adonnini Here are the APIs for to_backend.

Since you're using the partitioner, you should be doing

lowered: torch.fx.GraphModule = to_backend(to_be_lowered_module, XnnpackPartitioner())

or

lowered_program = edge_program.to_backend(XnnpackPartitioner())

@adonnini
Copy link
Author

@angelayi Thanks.

I had already tried to use lowered_program = edge_program.to_backend(XnnpackPartitioner())using the exported program instead of the edge program. Using your suggestion worked.

I also tried using
lowered: torch.fx.GraphModule = to_backend(to_be_lowered_module, XnnpackPartitioner())

it worked too.

Here is my code:
A

lowered_module: torch.fx.GraphModule = to_backend(to_be_lowered_module, XnnpackPartitioner())

B

lowered_module = edge_program.to_backend(XnnpackPartitioner())

However, unfortunately, in both cases execution failed when saving the file on disk with the following traceback:

Traceback (most recent call last):
  File "/home/adonnini1/Development/ContextQSourceCode/NeuralNetworks/trajectory-prediction-transformers-master/train-minimum.py", line 460, in <module>
    f.write(lowered_module.buffer())
AttributeError: 'EdgeProgramManager' object has no attribute 'buffer'

@angelayi
Copy link
Contributor

you need to do

lowered_module.to_executorch().buffer()

@adonnini
Copy link
Author

@angelayi Progress! with one small change to your suggestion,
lowered_module.to_executorch().buffer
Executuon completed successfully producing a .pte file of 87.2 MB
Now, as far as you know, is this model file usable for running inference on Android?
Thanks

@angelayi
Copy link
Contributor

I believe so: https://pytorch.org/executorch/stable/demo-apps-android.html

@adonnini
Copy link
Author

@angelayi I aleady went through the process described in
https://pytorch.org/executorch/stable/demo-apps-android.html
The last time I tried it failed for my app as it is tailored/hardwired for the executorch demo app.
I found a workaround but, then, the .pte file failed to load.
I'll try with this newly produced .pte and see what happens.
Thanks

@adonnini adonnini mentioned this issue Jan 15, 2024
Open
@adonnini
Copy link
Author

@angelayi should we consider this issue closed?
I attempted to run the model for inference from my Android app. Execution failed with the same error as described in #1350.
Thanks again for all your help

@angelayi
Copy link
Contributor

sounds good! :>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@angelayi angelayi

Labels
actionable Items in the backlog waiting for an appropriate impl/fix module: exir Issues related to Export IR and the code under exir/ triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@SS-JIA @larryliu0820 @angelayi @kimishpatel @adonnini