ORT aborts when loading the attached model

[justinchuby]

Describe the issue

resnet18.zip

When loading the model, inference session aborts without an informative error message.

To reproduce

import onnxruntime as ort

ort.InferenceSession("resnet18.onnx")

yeilds

/opt/rh/gcc-toolset-14/root/usr/include/c++/14/bits/stl_vector.h:1149: std::vector<_Tp, _Alloc>::const_reference std::vector<_Tp, _Alloc>::operator[](size_type) const [with _Tp = onnxruntime::NodeArg*; _Alloc = std::allocator<onnxruntime::NodeArg*>; const_reference = onnxruntime::NodeArg* const&; size_type = long unsigned int]: Assertion '__n < this->size()' failed.
fish: Job 1, 'python' terminated by signal SIGABRT (Abort)

Urgency

No response

Platform

Linux

OS Version

ubuntu

ONNX Runtime Installation

Released Package

ONNX Runtime Version or Commit ID

1.21.0

ONNX Runtime API

Python

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

[justinchuby]

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
onnxruntime::NodeArg::Name[abi:cxx11]() const (this=0x0) at /workspace/onnxruntime/onnxruntime/core/graph/graph.cc:222
222       return node_arg_info_.name();

[justinchuby]

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
onnxruntime::NodeArg::Name[abi:cxx11]() const (this=0x0) at /workspace/onnxruntime/onnxruntime/core/graph/graph.cc:222
222       return node_arg_info_.name();
(gdb) bt
#0  onnxruntime::NodeArg::Name[abi:cxx11]() const (this=0x0)
    at /workspace/onnxruntime/onnxruntime/core/graph/graph.cc:222
#1  0x00007ffff6df9120 in onnxruntime::graph_utils::NodeArgIsConstant (graph=..., node_arg=...)
    at /workspace/onnxruntime/onnxruntime/core/graph/graph_utils.cc:631
#2  0x00007ffff66259b4 in onnxruntime::(anonymous namespace)::selectors::GetLoneConsumerNode (node=..., 
    graph_viewer=...) at /workspace/onnxruntime/onnxruntime/core/optimizer/conv_activation_fusion.cc:31
#3  onnxruntime::(anonymous namespace)::selectors::ConvActivationSelector::Select (this=<optimized out>, 
    graph_viewer=..., node=...)
    at /workspace/onnxruntime/onnxruntime/core/optimizer/conv_activation_fusion.cc:88
#4  0x00007ffff65d3dc8 in onnxruntime::MatchAndProcess (save_context=0x0, selector_action_registry=..., 
    transformer_name=..., logger=..., modified=<optimized out>, node=..., graph_viewer=..., graph=...)
    at /workspace/onnxruntime/onnxruntime/core/optimizer/selectors_actions/selector_action_transformer.cc:112
#5  onnxruntime::SelectorActionTransformer::ApplySelectorsAndActions (this=<optimized out>, graph=..., 
    modified=<optimized out>, graph_level=0, logger=..., save_context=0x0)
    at /workspace/onnxruntime/onnxruntime/core/optimizer/selectors_actions/selector_action_transformer.cc:--Type <RET> for more, q to quit, c to continue without paging--c
200
#6  0x00007ffff65d749c in onnxruntime::SelectorActionTransformer::ApplyImpl (this=<optimized out>, graph=..., modified=<optimized out>, graph_level=<optimized out>, logger=...) at /workspace/onnxruntime/onnxruntime/core/optimizer/selectors_actions/selector_action_transformer.cc:318
#7  0x00007ffff654718b in onnxruntime::GraphTransformer::Apply (this=0x20f4ac0, graph=..., modified=@0x7fffffffcc77: false, logger=...) at /workspace/onnxruntime/onnxruntime/core/optimizer/graph_transformer.cc:14
#8  0x00007ffff6547c18 in onnxruntime::GraphTransformerManager::ApplyTransformers (this=this@entry=0x12e7a38, graph=..., level=level@entry=onnxruntime::TransformerLevel::Level2, logger=...) at /usr/include/c++/11/bits/unique_ptr.h:173
#9  0x00007ffff64d21ab in onnxruntime::InferenceSession::TransformGraph (this=0x12e7950, graph=..., saving_model_in_ort_format=<optimized out>) at /workspace/onnxruntime/onnxruntime/core/session/inference_session.cc:1402
#10 0x00007ffff64db5a8 in onnxruntime::InferenceSession::Initialize (this=this@entry=0x12e7950) at /workspace/onnxruntime/onnxruntime/core/session/inference_session.cc:2024
#11 0x00007ffff648ec21 in onnxruntime::python::InitializeSession(onnxruntime::InferenceSession*, std::function<void (onnxruntime::InferenceSession*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > > > const&)>, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::allocator<std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > > const&, std::unordered_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&) (sess=0x12e7950, ep_registration_fn=..., provider_types=std::vector of length 0, capacity 0, provider_options=..., disabled_optimizer_names=std::unordered_set with 0 elements) at /workspace/onnxruntime/onnxruntime/python/onnxruntime_pybind_state.cc:1369
#12 0x00007ffff649169e in operator() (disabled_optimizer_names=std::unordered_set with 0 elements, provider_options=std::vector of length 0, capacity 0, provider_types=std::vector of length 0, capacity 0, sess=0x97fcd0, __closure=0x12bc0e0) at /workspace/onnxruntime/onnxruntime/python/onnxruntime_pybind_state.cc:2052
#13 pybind11::detail::argument_loader<onnxruntime::python::PyInferenceSession*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::allocator<std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > > const&, std::unordered_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&>::call_impl<void, onnxruntime::python::addObjectMethods(pybind11::module&, onnxruntime::python::ExecutionProviderRegistrationFn)::<lambda(onnxruntime::python::PyInferenceSession*, const std::vector<std::__cxx11::basic_string<char> >&, const ProviderOptionsVector&, const std::unordered_set<std::__cxx11::basic_string<char> >&)>&, 0, 1, 2, 3, pybind11::detail::void_type> (f=..., this=0x7fffffffd610) at /workspace/onnxruntime/build/Linux/RelWithDebInfo/_deps/pybind11_project-src/include/pybind11/cast.h:1631
#14 pybind11::detail::argument_loader<onnxruntime::python::PyInferenceSession*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&, std::vector<std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >, std::allocator<std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > > const&, std::unordered_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&>::call<void, pybind11::detail::void_type, onnxruntime::python::addObjectMethods(pybind11::module&, onnxruntime::python::ExecutionProviderRegistrationFn)::<lambda(onnxruntime::python::PyInferenceSession*, const std::vector<std::__cxx11::basic_string<char> >&, const ProviderOptionsVector&, const std::unordered_set<std::__cxx11::basic_string<char> >&)>&> (f=..., this=0x7fffffffd610) at /workspace/onnxruntime/build/Linux/RelWithDebInfo/_deps/pybind11_project-src/include/pybind11/cast.h:1605
#15 operator() (__closure=0x0, call=...) at /workspace/onnxruntime/build/Linux/RelWithDebInfo/_deps/pybind11_project-src/include/pybind11/pybind11.h:279
#16 _FUN () at /workspace/onnxruntime/build/Linux/RelWithDebInfo/_deps/pybind11_project-src/include/pybind11/pybind11.h:249
#17 0x00007ffff63eacb6 in pybind11::cpp_function::dispatcher (self=<optimized out>, args_in=0x7fffa4a7e250, kwargs_in=0x0) at /workspace/onnxruntime/build/Linux/RelWithDebInfo/_deps/pybind11_project-src/include/pybind11/pybind11.h:971
#18 0x0000000000528767 in cfunction_call (func=0x7fffa4a5a930, args=<optimized out>, kwargs=<optimized out>) at /usr/local/src/conda/python-3.11.9/Objects/methodobject.c:542
#19 0x00000000005041ac in _PyObject_MakeTpCall (tstate=0x8a7a38 <_PyRuntime+166328>, callable=0x7fffa4a5a930, args=<optimized out>, nargs=<optimized out>, keywords=0x0) at /usr/local/src/conda/python-3.11.9/Objects/call.c:214
#20 0x00000000005116e7 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:4769
#21 0x0000000000538cc3 in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff7fb7080, tstate=0x8a7a38 <_PyRuntime+166328>) at /usr/local/src/conda/python-3.11.9/Include/internal/pycore_ceval.h:73
#22 _PyEval_Vector (kwnames=<optimized out>, argcount=2, args=0x7fffffffdc30, locals=0x0, func=<optimized out>, tstate=0x8a7a38 <_PyRuntime+166328>) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:6434
#23 _PyFunction_Vectorcall (func=<optimized out>, stack=0x7fffffffdc30, nargsf=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.11.9/Objects/call.c:393
#24 0x0000000000540022 in _PyObject_FastCallDictTstate (kwargs=<optimized out>, nargsf=<optimized out>, args=<optimized out>, callable=0x7fffa4a46660, tstate=0x8a7a38 <_PyRuntime+166328>) at /usr/local/src/conda/python-3.11.9/Objects/call.c:141
#25 _PyObject_Call_Prepend (kwargs=<optimized out>, args=<optimized out>, obj=0x7fffffffdc30, callable=0x7fffa4a46660, tstate=0x8a7a38 <_PyRuntime+166328>) at /usr/local/src/conda/python-3.11.9/Objects/call.c:482
#26 slot_tp_init (self=0x7fffffffdc30, args=<optimized out>, kwds=<optimized out>) at /usr/local/src/conda/python-3.11.9/Objects/typeobject.c:7855
#27 0x0000000000504173 in type_call (kwds=0x0, args=0x7ffff7bd4c10, type=<optimized out>) at /usr/local/src/conda/python-3.11.9/Objects/typeobject.c:1103
#28 _PyObject_MakeTpCall (tstate=0x8a7a38 <_PyRuntime+166328>, callable=0x12e5100, args=<optimized out>, nargs=<optimized out>, keywords=0x0) at /usr/local/src/conda/python-3.11.9/Objects/call.c:214
#29 0x00000000005116e7 in _PyEval_EvalFrameDefault (tstate=<optimized out>, frame=<optimized out>, throwflag=<optimized out>) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:4769
#30 0x00000000005cbeda in _PyEval_EvalFrame (throwflag=0, frame=0x7ffff7fb7020, tstate=0x8a7a38 <_PyRuntime+166328>) at /usr/local/src/conda/python-3.11.9/Include/internal/pycore_ceval.h:73
#31 _PyEval_Vector (tstate=0x8a7a38 <_PyRuntime+166328>, func=0x7ffff7bce020, locals=0x7ffff7bf64c0, args=<optimized out>, argcount=<optimized out>, kwnames=<optimized out>) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:6434
#32 0x00000000005cb5af in PyEval_EvalCode (co=<optimized out>, globals=0x7ffff7bf64c0, locals=<optimized out>) at /usr/local/src/conda/python-3.11.9/Python/ceval.c:1148
#33 0x00000000005ec6a7 in run_eval_code_obj (tstate=0x8a7a38 <_PyRuntime+166328>, co=0x7ffff7b80a80, globals=0x7ffff7bf64c0, locals=0x7ffff7bf64c0) at /usr/local/src/conda/python-3.11.9/Python/pythonrun.c:1741
#34 0x00000000005e8240 in run_mod (mod=<optimized out>, filename=<optimized out>, globals=0x7ffff7bf64c0, locals=0x7ffff7bf64c0, flags=<optimized out>, arena=<optimized out>) at /usr/local/src/conda/python-3.11.9/Python/pythonrun.c:1762
#35 0x00000000005fd192 in pyrun_file (fp=fp@entry=0x8eb370, filename=filename@entry=0x7ffff76ddcf0, start=start@entry=257, globals=globals@entry=0x7ffff7bf64c0, locals=locals@entry=0x7ffff7bf64c0, closeit=closeit@entry=1, flags=0x7fffffffe0c8) at /usr/local/src/conda/python-3.11.9/Python/pythonrun.c:1657
#36 0x00000000005fc55f in _PyRun_SimpleFileObject (fp=0x8eb370, filename=0x7ffff76ddcf0, closeit=1, flags=0x7fffffffe0c8) at /usr/local/src/conda/python-3.11.9/Python/pythonrun.c:440
#37 0x00000000005fc283 in _PyRun_AnyFileObject (fp=0x8eb370, filename=0x7ffff76ddcf0, closeit=1, flags=0x7fffffffe0c8) at /usr/local/src/conda/python-3.11.9/Python/pythonrun.c:79
#38 0x00000000005f6efe in pymain_run_file_obj (skip_source_first_line=0, filename=0x7ffff76ddcf0, program_name=0x7ffff7b52cd0) at /usr/local/src/conda/python-3.11.9/Modules/main.c:360
#39 pymain_run_file (config=0x88da80 <_PyRuntime+59904>) at /usr/local/src/conda/python-3.11.9/Modules/main.c:379
#40 pymain_run_python (exitcode=0x7fffffffe0c0) at /usr/local/src/conda/python-3.11.9/Modules/main.c:601
#41 Py_RunMain () at /usr/local/src/conda/python-3.11.9/Modules/main.c:680
#42 0x00000000005bbc79 in Py_BytesMain (argc=<optimized out>, argv=<optimized out>) at /usr/local/src/conda/python-3.11.9/Modules/main.c:734
#43 0x00007ffff7c29d90 in __libc_start_call_main (main=main@entry=0x5bbbd0 <main>, argc=argc@entry=2, argv=argv@entry=0x7fffffffe318) at ../sysdeps/nptl/libc_start_call_main.h:58
#44 0x00007ffff7c29e40 in __libc_start_main_impl (main=0x5bbbd0 <main>, argc=2, argv=0x7fffffffe318, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe308) at ../csu/libc-start.c:392
#45 0x00000000005bbac3 in _start ()

[justinchuby]

Check ValueInfoProto name?

[justinchuby]

Hi @skottmckay! I see that you are the last author for graph_utils::NodeArgIsConstant. Do you have an idea why the segfault is happening? (looks like the node arg doesn't have a name. Wondering what could cause it).

[edgchen1]

I found the issue.

onnxruntime/onnxruntime/core/optimizer/conv_activation_fusion.cc

Lines 28 to 34 in 8602d04

	const size_t input_edges_total = next_node->GetInputEdgesCount();
	int non_const_edges = 0;
	for (size_t edge_idx = 0; edge_idx < input_edges_total; ++edge_idx) {
	if (!graph_utils::NodeArgIsConstant(graph_viewer.GetGraph(), *next_node->InputDefs()[edge_idx])) {
	++non_const_edges;
	}
	}

If next_node has implicit inputs, as an If node might, input_edges_total will be larger than the number of explicit inputs which makes the indexing in *next_node->InputDefs()[edge_idx] eventually go past the end of the explicit input defs.

Working on a fix.

[justinchuby]

@edgchen1 Thanks! Scott helped with the analysis and pointed to the same issue. I wasn't able to make any code change yet so please feel free to work on it! For completeness I am reposting from Scott here:

there may be simpler ways as GetLoneConsumerNode probably shouldn't even be checking that edge.

• the name of the function would imply that input_edges_total for next_node should only be one
  • for the If node in question the count is 2. one for the explicit input and one for the implicit
• AFAIK there are no edges from initializers to nodes, so I don't think NodeArgIsConstant would ever return true
  • so unless something has changed there's no point calling NodeArgIsConstant with the src or dest index from an edge
• possibly the better fix is to check that input_edges_total == 1 and remove the (I believe) redundant call to NodeArgIsConstant